Has anybody used AWS SDK in MinDevkit?

Joe Kaplan's icon

Greetings,

I'd like to write a max external that moves files to and from an S3 bucket. AWS has a C++ SDK. They recommend acquiring it using the VCPKG package manager. I have vcpkg installed and I have run "vcpkg integrate install." So any libraries I've downloaded with vcpkg should be accessible to Visual Studio solutions.

If I create a new solution with Visual Studio everything checks out. I can include the AWS SDK headers, call the functions and write my code. But when I include the AWS SDK headers in a min-devkit project I suddenly get a bunch of linker errors.

Visual studio is finding the headers. The realtime error detection is not showing any problems. But things are failing to link when the code compiles.

If anybody has any suggestions for how to link libraries obtained from vcpkg with Min I would be very appreciative. Thank you!

Rob Ramirez's icon
Joe Kaplan's icon

I'm pulling my hair out here. This is what I've accomplished:

I followed the vcpkg Walkthrough that Rob linked to above. I was able to get it to work. I downloaded sqlite3 via vcpkg and was able to bring sqlite3 into my Min project.

The walkthrough for AWS SDK is extremely similar. I am able to generate the project successfully, but when I try to build I get linker errors. 13 unresolved externals to be precise.

I don't know what to do. I can build the AWS SDK as a standalone console app. I can bring a different library into Min. Cmake is not indicating any problems.

The only difference between the sqlite example and the AWS example is that the AWS example has a couple more lines added to cmakelists.txt. I tried adding/removing various lines and placing them in different orders. I was able to generate different cmake errors this way, but I was not able to get anything to build. In one configuration, I was also able to generate an additional build time error about being unable to locate a header header file. I don't know if that's significant.

Below is cmakelists.txt to generate a console app for AWS and the default Min project cmakelists.txt. If anybody has any suggestions for how to merge the two... or any other ideas to try... I would be truly overjoyed. Thanks!

# Minimal CMakeLists.txt for the AWS SDK for C++.
cmake_minimum_required(VERSION 3.3)
set(CMAKE_CXX_STANDARD 11)
project(app LANGUAGES CXX)

# Use shared libraries
set(BUILD_SHARED_LIBS ON CACHE STRING "Link to shared libraries by default.")

#Include in any AWS service packages your code will be using by service name
find_package(AWSSDK REQUIRED COMPONENTS s3)
add_executable(${PROJECT_NAME} "main.cpp") #add app's main sourcefile

set_compiler_flags(${PROJECT_NAME})
set_compiler_warnings(${PROJECT_NAME})
target_link_libraries(${PROJECT_NAME} PUBLIC ${AWSSDK_LINK_LIBRARIES})

(note that you have to specify "PUBLIC" in target_link_libraries here, since that is how it is used elsewhere in Max SDK)

# Copyright 2018 The Min-DevKit Authors. All rights reserved.
# Use of this source code is governed by the MIT License found in the License.md file.

cmake_minimum_required(VERSION 3.0)

set(C74_MIN_API_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../min-api)
include(${C74_MIN_API_DIR}/script/min-pretarget.cmake)

#############################################################
# MAX EXTERNAL
#############################################################

include_directories( 
    "${C74_INCLUDES}"
)

set( SOURCE_FILES
    ${PROJECT_NAME}.cpp
    ${MAX_SDK_INCLUDES}/common/commonsyms.c
)

add_library( 
    ${PROJECT_NAME} 
    MODULE
    ${SOURCE_FILES}
)

include(${C74_MIN_API_DIR}/script/min-posttarget.cmake)

#############################################################
# UNIT TEST
#############################################################

include(${C74_MIN_API_DIR}/test/min-object-unittest.cmake)

Joe Kaplan's icon

Hi,

I've spent ten days crunching on this and seeking help from others in my network. I think I understand a little bit more about what's going on but I'm no closer to a solution. If anyone has any suggestions for things to try it would be a big help.

I'm getting 6 warnings and 14 errors. They all look similar. Below are some examples.

The unresolved externals have to do with functions in the namespace std::
To me this suggests the problem is not with linking to the AWS libraries per se, but something about the way AWS and MIN interact is messing up standard c++ functions.

I don't know how to troubleshoot this without beginning to reverse engineer both Aws-SDK and MIN. Neither of which are viable options for me. As far as I can tell, I ought to be able to bring the AWS C++ SDK into MIN and make cool stuff even cooler!

Below is a sample repo and steps to reproduce. I have also raised an issue over on the aws-sdk-cpp github. But If anybody here has any suggestions I would be very appreciative.

Reproduction Steps

Install AWS from VCPKG or from Source

Repro steps:
Clone Repo
In repo directory:
md build
cd build

If you installed from vcpkg:
cmake "-DCMAKE_TOOLCHAIN_FILE=[PATH TO YOUR TOOLCHAIN FILE]" -G "Visual Studio 17 2022" ..

If you installed from source:
Add the addendum below to "./source/projects/AWSTest3.hello-world/CMakeLists.txt" then:
cmake "-DCMAKE_TOOLCHAIN_FILE=[PATH TO YOUR TOOLCHAIN FILE]" -G "Visual Studio 17 2022" ..

open awstest3.sln
In Solution Explorer, right click "awstest3.hello_world"
Select Build
View Errors

Cmakelists addendum for AWS Source Builds

#Set the location of where Windows can find the installed libraries of the SDK.
if(MSVC)
string(REPLACE ";" "/aws-cpp-sdk-all;" SYSTEM_MODULE_PATH "${CMAKE_SYSTEM_PREFIX_PATH}/aws-cpp-sdk-all")
list(APPEND CMAKE_PREFIX_PATH ${SYSTEM_MODULE_PATH})
endif()

MakePatchesNotWar's icon

"If anyone has any suggestions for things to try it would be a big help."

Saw your post before but didn't responde earlier. Do you really need this as an external? I ask because there is a library? (or whatever its called in javascript) for node.js and it might be easier to explore that?

Wish i could help you with the devkit but alas....

Rob Ramirez's icon

did you add the line set(CMAKE_CXX_STANDARD 11) ? That might be relevant to your errror message.

Joe Kaplan's icon

@Luvulongtime
Thanks for the suggestion! Moving assets to and from S3 is one component of what I'm hoping to achieve with these externals. Having to handle AWS interactions in a separate node.script object would make the whole package a little unwieldy in Max compared to the idiomatic flow I'm aiming for. Also, I have concerns about writing credential management code in an open source format. I appreciate the thought though. Switching to "Node for Max" for this component could potentially be a temporary workaround that would allow me to work on other parts of the project if I get blocked here for too long.

@Rob Ramirez
Thanks! I tried adding set(CMAKE_CXX_STANDARD 11) immediately following cmake_minimum_required(VERSION 3.0) in cmakelists.txt in the project folder for the object, in the root folder for the package, and both. It did not have any impact on the error.

Is it possible that some component deeper in Min or Max API is setting a different version and overwriting that instruction?

Joe Kaplan's icon

I was able to meet up with a friend and we made some good progress on this. It appears that Max is unhappy with the way AWS is linking to its dynamic libraries. I found this set of preprocessor macros in one of the AWS headers. If I #undef USE_IMPORT_EXPORT we can skip preprocessor block that defines __declspec(dllimport) for the AWS functions and then we are off to the races. The external builds. I can call AWS functions. It works! I don't know why it works. It seems like it probably shouldn't work. But it works.

If we tell the computer that the core AWS functions are not dllimports, but still provide the DLLs, it will go ahead and import them anyway. Why does Max care in the first place?!


Manually modifying the AWS headers is clearly not the right solution. Ultimately, I'd like to statically link the library anyway. I really don't want Amazon's dozen or so .dlls to have to travel with my package . So I rebuilt the AWS SDK from source as a static library.

Now I have a new problem. I have to link a bunch of static libs into the Min project which is required to be built as dynamic library.

After linking up the static libraries I get these errors:

some elements are configurated for dynamic release, others are configured for static release.

If I change the project properties to Configuration type .lib then the external will build.

I changed configuration type from .dll to .lib and the project compiled successfully.


However, it doesn't work. Max gets real angry when I try to make an instance of my object.

Max gets real angry when I try to use the external configured as a .lib.

So the configuration for the project has to be a .dll.

But we've arrived at a legitimate and answerable question I think?

How can I configure the Min Devkit project so that I can include static links to other libraries, but still compile it as the .dll Max needs it to be?


Rob Ramirez's icon

a shot in the dark, but maybe try setting this FORCE_SHARED_CRT cmake option to ON when building the static AWS lib - https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/cmake-params.html#cmake-force-shared-crt

If that doesn't work, you can try changing the CRT linking in the of the max external. This is set by the cmake scripts, and you can try changing /MT and /MTd to /MD and /MDd


you definitely can't mess with the max external configuration type.

Joe Kaplan's icon

https://stackoverflow.com/questions/14714877/mismatch-detected-for-runtimelibrary

I found this probably just as you were posting, Rob. Basically suggesting a very similar solution. I got it working!!! Looks like there may be a couple different ways to handle it. I will explore the options you mention above and use the one that seems best for my build. Thanks!