Hi, I think I figured out why this is happening. The old generator code expected one or more subdirectories/namespaces within the top-level folder. This archive consists of one top-level folder acting as the namespace folder itself, which the backend couldn’t deal with.
Unrelated possible issue I ran into: Nginx limits file upload size to 1M (I think total) by default. Is this an issue? I can bump it up in deployment/nginx.conf if needed.
A .zip with all the public regulated data types is 144 KiB large. 1 MiB should be sufficient but in the interest of future-proofing the configuration we might want to make it 10 MiB, just in case.
It’s working well for me. I just tested the deployed app at nunaweb.uavcan.org and got my .zip back successfully. It appears to be valid.
I have created a new branch called cicd in which I have made changes to “Dockerize” the application so that it can be used for CI/CD in production. There is now a separate container for the nunaserver, the nunaweb and the nunaproxy. The proxy will handle all the requests and direct them to the relevant components. This eliminates the need for port 8000 to be exposed for the API.
The main issue with the app at present is that docker-compose uses volumes to access data across containers. Each component needs to be encapsulated so that the containers can be deployed individually of each other.
In particular, the nunaserver component uses gunicorn to serve the web app data and shares a volume with the proxy in order serve the static content. The nunaserver container needs to serve both. The choices are either a) have gunicorn serve the static content; or b) replace gunicorn with nginx and have it serve both.
The volumes actually share data across 3 containers: the Celery container, as well as the nunaserver and nunaproxy containers. The problem is that we need to share data across Celery and nunaserver, otherwise the application can’t function properly. I do agree with you on replacing gunicorn with nginx and having nginx serve the application though.
How do you propose we solve this? Moreover, what specific issues does the volume data sharing cause?
Containers need to be self-contained. The usual way to share data between services/containers is via the network interface rather than the file system. I suppose there are two options: 1) put Celery and nunaserver in the same container; or 2) share the data between the two services via a network call.
I have separated the proxy container from the others. I am not sure what data it needs from the other services. It seemed to only require the file system to serve static data requests. Is there something I am missing?
I have uploaded the images to DockerHub and updated the docker-compose.yml file to pull from those images. After running docker-compose up from the deployment directory, everything appears to work except I am not able to download the .zip file (although I have not checked uploads). It seems to create the .zip file in the Celery container, rather than the nunaserver as I thought, so perhaps we can just get Celery to serve these http requests if this is the only outstanding issue. Perhaps you can give the code in cicd branch a try in your dev environment and let me know if the Docker builds are correct.
The proxy container only requires the static files to serve the static file requests. You’ll see another mount in there for the configuration; this is for convenience, as the nginx.conf inside the deployment folder can be easily edited without having to regenerate the image. If you want to bake it in instead, that’s your choice.
A few things I want to clarify:
I am fairly sure that having filesystem mounts for a container is normal and perfectly acceptable. Many docker-compose deployed applications use this approach - some use it for data persistence (e.g. databases), while others use it for sharing files between containers; either way, it should be completely fine, as these containers will generally run as part of the whole docker-compose configuration, not completely standalone. I understand there are some scalability issues with running with filesystem mounts; I’m thinking of integrating a service such as Minio for that, but filesystem mount should be acceptable for now (shouldn’t be any significant security risk or anything like that).
Sharing the data between the two services via network call is a possibility, won’t this increase the network overhead of the application, especially if it’s transferring medium to large files from Flask to Celery? I get, however, that it’s a cleaner solution; see proposed minio integration above.
Putting Celery and nunaserver in the same container is…very jank, to say the least. Each container runs a single service; while it’s possible to run multiple services in the same container, the ways to do that are hacky and generally not considered great practice, so I’d rather not.
Celery is a task queue implementation. It does not serve anything.
TL;DR: The filesystem mount configuration as it is right now should function fine for our purposes at the moment. If we want more stability/a cleaner architecture, we can integrate an object storage service such as minio to hold uploaded files and Celery build artifacts.
I will give the code a try and check that the Docker builds are correct.
One thing to note: Docker-Compose can handle building as well (https://docs.docker.com/compose/compose-file/compose-file-v3/#build). If you add this to the docker-compose configuration it will be able to build all necessary containers on run and even push them with docker-compose build && docker-compose push. Can you add this?
I am fairly sure that having filesystem mounts for a container is normal and perfectly acceptable.
As explained in Step 5 of the documentation example, the volume feature allows “you to modify the code on the fly, without having to rebuild the image” and that “This mode should only be used in development.”
You probably need to make changes to your app configuration to make it ready for production. These changes may include:
Removing any volume bindings for application code, so that code stays inside the container and can’t be changed from outside
Binding to different ports on the host
Setting environment variables differently, such as reducing the verbosity of logging, or to specify settings for external services such as an email server
Specifying a restart policy like restart: always to avoid downtime
Adding extra services such as a log aggregator
It makes sense because we want to freeze the build and prevent changes to the code in the production environment. All the source code and application should be inside the container. I could see an exception perhaps where you might want to share data between the host and the container (e.g. maybe with SSL certificates) but not between services.
Sharing the data between the two services via network call is a possibility, won’t this increase the network overhead of the application, especially if it’s transferring medium to large files from Flask to Celery? I get, however, that it’s a cleaner solution; see proposed minio integration above.
The services are connected via a network bridge which is a just socket on the host so data transfers should be fast. But we should not assume that services will be on the same host. It should work fine (but more slowly) with the service components on separate hosts.
Putting Celery and nunaserver in the same container is…very jank, to say the least. Each container runs a single service; while it’s possible to run multiple services in the same container, the ways to do that are hacky and generally not considered great practice, so I’d rather not.
Agreed. I am not sure what the Celery best practices are but general single-task containers are best.
Celery is a task queue implementation. It does not serve anything.
Why does the .zip file generated by the app for download appear in the /nunaserver/static/ directory in the Celery container? Should this not be returned to nunaserver for it to serve back to the client?
One thing to note: Docker-Compose can handle building as well (https://docs.docker.com/compose/compose-file/compose-file-v3/#build). If you add this to the docker-compose configuration it will be able to build all necessary containers on run and even push them with docker-compose build && docker-compose push . Can you add this?
The goal here is to create a CI/CD workflow. GitHub Actions will take care of the build and create new Docker images for DockerHub when new code is checked into the GitHub repository (see .github/workflows/production.yml). Docker Compose can be used to pull the latest images and restart the services so that those changes appear in production. This way, no source code is needed in production; just pull the images and run it with the environment variables in the docker-compose.yml file.
You might need a separate docker-compose.yml file for development; or, as shown in the docker-compose documentation example, use an environment variable to indicate which environment you are using and build accordingly.
I think there might be a misunderstanding here. The application code itself is not mounted and thus cannot be changed from outside. The only volume mount is the static directory, which should be acceptable at small scale, but I do acknowledge that it’s not very scalable and thus not best practice. I’m working on integrating Minio so we can solve this issue.
Why does the .zip file generated by the app for download appear in the /nunaserver/static/ directory in the Celery container? Should this not be returned to nunaserver for it to serve back to the client?
As far as I can recall, there’s no one easy mechanism for Celery to “return” a file artifact such as our generated zips back to Flask. Furthermore, Flask is not optimal for serving static files anyway; while it’s capable of the job, production webservers (e.g. nginx) can do it more efficiently.
This problem is again solved by minio, which is able to serve stored files. A simple reverse proxy connection will allow minio to serve the generated artifacts. We can have Celery upload the generated zip once it is completed.
I think there might be a misunderstanding here. The application code itself is not mounted and thus cannot be changed from outside. The only volume mount is the static directory,
Apologies, I was incorrect. This was how it was supposed to be mounted, but due to an oversight the entire application code was mounted.
Updates: the project now uses minio for object storage, so that eliminates the need for mounts. Deployment files are updated in main.
I have completed the updates for production. It now runs off the secure port (443) with a Let’s Encrypt certificate and all services in Docker containers. GitHub Actions builds and deploys changes to production from the cicd branch (for the moment).
It would be good if someone could verify that the application works as expected from the new secure port URL.
Are these changes complete and ready for production? It looks like the configuration settings are for testing. It might have been better to put them in a feature branch for review as I cannot merge the cicd branch into main until we resolve the conflicts with nginx.conf and prod.docker-compose.yml. Perhaps we should protect main to enforce pull requests. What do you think?
The application does not seem to be functioning correctly but I am not sure if the culprit is in the deployment configuration or in the application itself. These are the inputs I supplied to the application (where my_project.zip is the same I posted here earlier):
When I try to compile just my_project.zip, I click “Submit” and nothing happens. No logs on the server either. I tried with different browsers and settings without success. It might be the my_project.zip file. The SHA256 checksum for it is: 68a1db86e9ee4591f13d1bcb282868602a8ff5bd67996bf96f8be218b9e5729c.
I am not sure either. Perhaps @bbworld1 could clarify.
Perhaps we should protect main to enforce pull requests. What do you think?
I agree, this is a good idea. Minio should probably have gone into a feature branch first.
I did test the (old) production configuration with minio and it works properly - I can upload project zips and get generated code back. Perhaps we can integrate it into the new cicd production configuration?
It looks like the configuration settings are for testing.
I may have missed something but the configuration looks valid to me. Is there some configuration that needs to be done to make it production ready (besides setting the right production urls and values)?
The application does not seem to be functioning correctly but I am not sure if the culprit is in the deployment configuration or in the application itself.
Is this a new deployment from main or from cicd? I recall the application worked earlier.
I don’t believe deploy on push to main is actually set up yet, and the minio port was tested working, which leads me to believe it was a change in the deployment configuration in the cicd branch - not sure what caused this yet.
@pavel.kirienko The error is caused by a deployment change. The cicd docker-compose configuration omitted the /tmp mount, which was needed as temporary working directories needed to be accessed by both Flask and Celery.
This issue has been fixed with the introduction of minio - everything now happens in Celery, and thus this mount is not needed anymore. That being said, we have to integrate the changes before this will work, so we should do that ASAP.
I have rebased cicd branch on top of the main branch and resolved the file conflicts. It now deploys to production on pushes to main. I still need modify the GH Actions build to set the minio password in the docker-compose.yml file. We don’t want to set in the file and checked into GH.
Could we please test this new deployment again? If minio is working, it should resolve the namespace issues @pavel.kirienko mentioned earlier.
Okay, yes, the zip is now handled properly but we are still missing nunavut/serialization. This is probably related to the application itself, after all. @bbworld1 when you run it locally, does it generate the support library correctly?
