# An easy-to-use web service for compiling DSDL **URL:** \ **Category:** Nunavut\ **Created:** [January 13, 2021, 1:40pm UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074 "2021-01-13T13:40:27Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![bbworld1](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/bbworld1/32/786_2.png) [@bbworld1](https://forum.opencyphal.org/u/bbworld1)\ **Post date:** [February 4, 2021, 3:45am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/21 "2021-02-04T03:45:56Z")
Hi, I think I figured out why this is happening. The old generator code expected one or more subdirectories/namespaces within the top-level folder. This archive consists of one top-level folder acting as the namespace folder itself, which the backend couldn’t deal with. I’ve pushed and merged a fix ([https://github.com/UAVCAN/nunaweb/pull/3](https://github.com/UAVCAN/nunaweb/pull/3)). Unrelated possible issue I ran into: Nginx limits file upload size to 1M (I think total) by default. Is this an issue? I can bump it up in deployment/nginx.conf if needed. ---
**Author:** ![clydej](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/clydej/32/1098_2.png) [@clydej](https://forum.opencyphal.org/u/clydej)\ **Post date:** [February 4, 2021, 8:02am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/22 "2021-02-04T08:02:38Z")
I pulled in the code and redeployed the app but I see no change in the behavior. After adding the zip file and clicking “submit”, nothing happens. I will leave the upload size limit question to users of the app. ---
**Author:** ![bbworld1](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/bbworld1/32/786_2.png) [@bbworld1](https://forum.opencyphal.org/u/bbworld1)\ **Post date:** [February 4, 2021, 8:52am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/23 "2021-02-04T08:52:04Z")
![image](https://forum.opencyphal.org/uploads/default/original/1X/0ecfde027727a6fdf483d86c583c262da8a67d79.png) It appears to work fine for me. Maybe just a caching thing? Although I don’t think nunaweb uses any sort of cache that would cause an error. ---
**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\ **Post date:** [February 4, 2021, 10:12am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/24 "2021-02-04T10:12:14Z")
> [@bbworld1](#): > > Nginx limits file upload size to 1M (I think total) by default. Is this an issue? I can bump it up in deployment/nginx.conf if needed. A .zip with all the public regulated data types is 144 KiB large. 1 MiB should be sufficient but in the interest of future-proofing the configuration we might want to make it 10 MiB, just in case. > [@clydej](#): > > I pulled in the code and redeployed the app but I see no change in the behavior. After adding the zip file and clicking “submit”, nothing happens. It’s working well for me. I just tested the deployed app at [nunaweb.uavcan.org](http://nunaweb.uavcan.org) and got my .zip back successfully. It appears to be valid. ---
**Author:** ![clydej](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/clydej/32/1098_2.png) [@clydej](https://forum.opencyphal.org/u/clydej)\ **Post date:** [February 16, 2021, 1:59pm UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/25 "2021-02-16T13:59:11Z")
I have created a new branch called `cicd` in which I have made changes to “Dockerize” the application so that it can be used for CI/CD in production. There is now a separate container for the `nunaserver`, the `nunaweb` and the `nunaproxy`. The proxy will handle all the requests and direct them to the relevant components. This eliminates the need for port 8000 to be exposed for the API. The main issue with the app at present is that docker-compose uses `volumes` to access data across containers. Each component needs to be encapsulated so that the containers can be deployed individually of each other. In particular, the `nunaserver` component uses `gunicorn` to serve the web app data and shares a volume with the proxy in order serve the static content. The `nunaserver` container needs to serve both. The choices are either a) have `gunicorn` serve the static content; or b) replace `gunicorn` with `nginx` and have it serve both. Can we get this changed on the application side? ---
**Author:** ![bbworld1](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/bbworld1/32/786_2.png) [@bbworld1](https://forum.opencyphal.org/u/bbworld1)\ **Post date:** [February 16, 2021, 4:32pm UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/26 "2021-02-16T16:32:30Z")
The volumes actually share data across 3 containers: the Celery container, as well as the `nunaserver` and `nunaproxy` containers. The problem is that we need to share data across Celery and `nunaserver`, otherwise the application can’t function properly. I do agree with you on replacing `gunicorn` with `nginx` and having `nginx` serve the application though. How do you propose we solve this? Moreover, what specific issues does the volume data sharing cause? ---
**Author:** ![clydej](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/clydej/32/1098_2.png) [@clydej](https://forum.opencyphal.org/u/clydej)\ **Post date:** [February 17, 2021, 1:16am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/27 "2021-02-17T01:16:58Z")
Containers need to be self-contained. The usual way to share data between services/containers is via the network interface rather than the file system. I suppose there are two options: 1) put Celery and `nunaserver` in the same container; or 2) share the data between the two services via a network call. I have separated the proxy container from the others. I am not sure what data it needs from the other services. It seemed to only require the file system to serve static data requests. Is there something I am missing? I have uploaded the images to DockerHub and updated the `docker-compose.yml` file to pull from those images. After running `docker-compose up` from the `deployment` directory, everything appears to work except I am not able to download the .zip file (although I have not checked uploads). It seems to create the .zip file in the Celery container, rather than the `nunaserver` as I thought, so perhaps we can just get Celery to serve these http requests if this is the only outstanding issue. Perhaps you can give the code in `cicd` branch a try in your dev environment and let me know if the Docker builds are correct. ---
**Author:** ![bbworld1](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/bbworld1/32/786_2.png) [@bbworld1](https://forum.opencyphal.org/u/bbworld1)\ **Post date:** [February 17, 2021, 3:23am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/28 "2021-02-17T03:23:32Z")
The proxy container only requires the static files to serve the static file requests. You’ll see another mount in there for the configuration; this is for convenience, as the nginx.conf inside the deployment folder can be easily edited without having to regenerate the image. If you want to bake it in instead, that’s your choice. A few things I want to clarify: 1. I am fairly sure that having filesystem mounts for a container is normal and perfectly acceptable. Many docker-compose deployed applications use this approach - some use it for data persistence (e.g. databases), while others use it for sharing files between containers; either way, it should be completely fine, as these containers will generally run as part of the whole docker-compose configuration, not completely standalone. I understand there are some scalability issues with running with filesystem mounts; I’m thinking of integrating a service such as Minio for that, but filesystem mount should be acceptable for now (shouldn’t be any significant security risk or anything like that). 1. Sharing the data between the two services via network call is a possibility, won’t this increase the network overhead of the application, especially if it’s transferring medium to large files from Flask to Celery? I get, however, that it’s a cleaner solution; see proposed minio integration above. 2. Putting Celery and nunaserver in the same container is…very jank, to say the least. Each container runs a single service; while it’s possible to run multiple services in the same container, the ways to do that are hacky and generally not considered great practice, so I’d rather not. 3. Celery is a task queue implementation. It does not serve anything. TL;DR: The filesystem mount configuration as it is right now should function fine for our purposes at the moment. If we want more stability/a cleaner architecture, we can integrate an object storage service such as minio to hold uploaded files and Celery build artifacts. I will give the code a try and check that the Docker builds are correct. One thing to note: Docker-Compose can handle building as well ([https://docs.docker.com/compose/compose-file/compose-file-v3/#build](https://docs.docker.com/compose/compose-file/compose-file-v3/#build)). If you add this to the docker-compose configuration it will be able to build all necessary containers on run and even push them with `docker-compose build && docker-compose push`. Can you add this? ---
**Author:** ![clydej](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/clydej/32/1098_2.png) [@clydej](https://forum.opencyphal.org/u/clydej)\ **Post date:** [February 17, 2021, 10:01am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/29 "2021-02-17T10:01:03Z")
> 1. I am fairly sure that having filesystem mounts for a container is normal and perfectly acceptable. As explained in [Step 5](https://docs.docker.com/compose/gettingstarted/#step-5-edit-the-compose-file-to-add-a-bind-mount) of the documentation example, the `volume` feature allows “you to modify the code on the fly, without having to rebuild the image” and that “This mode should only be used in development.” The documentation about using [docker-compose in production](https://docs.docker.com/compose/production/), it says: > You probably need to make changes to your app configuration to make it ready for production. These changes may include: > > - Removing any volume bindings for application code, so that code stays inside the container and can’t be changed from outside > - Binding to different ports on the host > - Setting environment variables differently, such as reducing the verbosity of logging, or to specify settings for external services such as an email server > - Specifying a restart policy like `restart: always` to avoid downtime > - Adding extra services such as a log aggregator It makes sense because we want to freeze the build and prevent changes to the code in the production environment. All the source code and application should be inside the container. I could see an exception perhaps where you might want to share data between the host and the container (e.g. maybe with SSL certificates) but not between services. > 1. Sharing the data between the two services via network call is a possibility, won’t this increase the network overhead of the application, especially if it’s transferring medium to large files from Flask to Celery? I get, however, that it’s a cleaner solution; see proposed minio integration above. The services are connected via a network bridge which is a just socket on the host so data transfers should be fast. But we should not assume that services will be on the same host. It should work fine (but more slowly) with the service components on separate hosts. > 1. Putting Celery and nunaserver in the same container is…very jank, to say the least. Each container runs a single service; while it’s possible to run multiple services in the same container, the ways to do that are hacky and generally not considered great practice, so I’d rather not. Agreed. I am not sure what the Celery best practices are but general single-task containers are best. > 1. Celery is a task queue implementation. It does not serve anything. Why does the .zip file generated by the app for download appear in the `/nunaserver/static/` directory in the Celery container? Should this not be returned to `nunaserver` for it to serve back to the client? > One thing to note: Docker-Compose can handle building as well ([Legacy versions | Docker Docs](https://docs.docker.com/compose/compose-file/compose-file-v3/#build)). If you add this to the docker-compose configuration it will be able to build all necessary containers on run and even push them with `docker-compose build && docker-compose push` . Can you add this? The goal here is to create a CI/CD workflow. GitHub Actions will take care of the build and create new Docker images for DockerHub when new code is checked into the GitHub repository (see `.github/workflows/production.yml`). Docker Compose can be used to pull the latest images and restart the services so that those changes appear in production. This way, no source code is needed in production; just pull the images and run it with the environment variables in the `docker-compose.yml` file. You might need a separate `docker-compose.yml` file for development; or, as shown in [the docker-compose documentation example](https://docs.docker.com/compose/gettingstarted/#step-5-edit-the-compose-file-to-add-a-bind-mount), use an environment variable to indicate which environment you are using and build accordingly. ---
**Author:** ![bbworld1](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/bbworld1/32/786_2.png) [@bbworld1](https://forum.opencyphal.org/u/bbworld1)\ **Post date:** [February 18, 2021, 7:58am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/30 "2021-02-18T07:58:36Z")
> “This mode should only be used in development.” I think there might be a misunderstanding here. The application code itself is not mounted and thus cannot be changed from outside. The only volume mount is the `static` directory, which should be acceptable at small scale, but I do acknowledge that it’s not very scalable and thus not best practice. I’m working on integrating Minio so we can solve this issue. > Why does the .zip file generated by the app for download appear in the `/nunaserver/static/` directory in the Celery container? Should this not be returned to `nunaserver` for it to serve back to the client? As far as I can recall, there’s no one easy mechanism for Celery to “return” a file artifact such as our generated zips back to Flask. Furthermore, Flask is not optimal for serving static files anyway; while it’s capable of the job, production webservers (e.g. nginx) can do it more efficiently. This problem is again solved by minio, which is able to serve stored files. A simple reverse proxy connection will allow minio to serve the generated artifacts. We can have Celery upload the generated zip once it is completed. ---
**Author:** ![bbworld1](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/bbworld1/32/786_2.png) [@bbworld1](https://forum.opencyphal.org/u/bbworld1)\ **Post date:** [February 18, 2021, 7:02pm UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/31 "2021-02-18T19:02:28Z")
> I think there might be a misunderstanding here. The application code itself is not mounted and thus cannot be changed from outside. The only volume mount is the `static` directory, Apologies, I was incorrect. This was how it was _supposed_ to be mounted, but due to an oversight the entire application code was mounted. Updates: the project now uses minio for object storage, so that eliminates the need for mounts. Deployment files are updated in main. ---
**Author:** ![clydej](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/clydej/32/1098_2.png) [@clydej](https://forum.opencyphal.org/u/clydej)\ **Post date:** [February 19, 2021, 1:36am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/32 "2021-02-19T01:36:46Z")
I have completed the updates for production. It now runs off the secure port (443) with a Let’s Encrypt certificate and all services in Docker containers. GitHub Actions builds and deploys changes to production from the `cicd` branch (for the moment). It would be good if someone could verify that the application works as expected from the new [secure port URL](https://nunaweb.uavcan.org). > [@bbworld1](#): > > Updates: the project now uses minio for object storage, so that eliminates the need for mounts. Deployment files are updated in main. Are these changes complete and ready for production? It looks like the configuration settings are for testing. It might have been better to put them in a feature branch for review as I cannot merge the `cicd` branch into `main` until we resolve the conflicts with `nginx.conf` and `prod.docker-compose.yml`. Perhaps we should protect `main` to enforce pull requests. What do you think? ---
**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\ **Post date:** [February 19, 2021, 1:48am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/33 "2021-02-19T01:48:59Z")
> [@clydej](#): > > It would be good if someone could verify that the application works as expected from the new [secure port URL](https://nunaweb.uavcan.org). The application does not seem to be functioning correctly but I am not sure if the culprit is in the deployment configuration or in the application itself. These are the inputs I supplied to the application (where `my_project.zip` is the same I posted here earlier): ![image](https://forum.opencyphal.org/uploads/default/original/1X/efde6b5f2038d71d8f33cff195b8dcdf61b0d385.png) The expected output is an archive with four directories inside: - **`uavcan`** (the standard regulated namespace from GitHub) - **`reg`** (a non-standard regulated namespace from GitHub) - **`my_project`** (an unregulated namespace from the archive) - **`nunavut`** (the support library generated by Nunavut) The archive I got contains only two namespaces: `uavcan` and `reg`. The code generated in these directories is, however, correct. If I compile just `my_project.zip`, I get a file not found error: ![image](https://forum.opencyphal.org/uploads/default/original/1X/6abe72e6636f43d3f7971e76103f5779f04a96e2.png) ---
**Author:** ![clydej](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/clydej/32/1098_2.png) [@clydej](https://forum.opencyphal.org/u/clydej)\ **Post date:** [February 19, 2021, 2:21am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/34 "2021-02-19T02:21:26Z")
> [@pavel.kirienko](#): > > If I compile just `my_project.zip` , I get a file not found error: When I try to compile just `my_project.zip`, I click “Submit” and nothing happens. No logs on the server either. I tried with different browsers and settings without success. It might be the `my_project.zip` file. The SHA256 checksum for it is: `68a1db86e9ee4591f13d1bcb282868602a8ff5bd67996bf96f8be218b9e5729c`. > [@pavel.kirienko](#): > > The application does not seem to be functioning correctly but I am not sure if the culprit is in the deployment configuration or in the application itself. I am not sure either. Perhaps @bbworld1 could clarify. ---
**Author:** ![bbworld1](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/bbworld1/32/786_2.png) [@bbworld1](https://forum.opencyphal.org/u/bbworld1)\ **Post date:** [February 19, 2021, 4:16am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/35 "2021-02-19T04:16:17Z")
> Perhaps we should protect `main` to enforce pull requests. What do you think? I agree, this is a good idea. Minio should probably have gone into a feature branch first. I did test the (old) production configuration with minio and it works properly - I can upload project zips and get generated code back. Perhaps we can integrate it into the new cicd production configuration? > It looks like the configuration settings are for testing. I may have missed something but the configuration looks valid to me. Is there some configuration that needs to be done to make it production ready (besides setting the right production urls and values)? > The application does not seem to be functioning correctly but I am not sure if the culprit is in the deployment configuration or in the application itself. Is this a new deployment from main or from cicd? I recall the application worked earlier. I don’t believe deploy on push to main is actually set up yet, and the minio port was tested working, which leads me to believe it was a change in the deployment configuration in the cicd branch - not sure what caused this yet. ---
**Author:** ![bbworld1](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/bbworld1/32/786_2.png) [@bbworld1](https://forum.opencyphal.org/u/bbworld1)\ **Post date:** [February 19, 2021, 5:42am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/36 "2021-02-19T05:42:09Z")
@pavel.kirienko The error is caused by a deployment change. The cicd docker-compose configuration omitted the /tmp mount, which was needed as temporary working directories needed to be accessed by both Flask and Celery. This issue has been fixed with the introduction of minio - everything now happens in Celery, and thus this mount is not needed anymore. That being said, we have to integrate the changes before this will work, so we should do that ASAP. ---
**Author:** ![clydej](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/clydej/32/1098_2.png) [@clydej](https://forum.opencyphal.org/u/clydej)\ **Post date:** [February 19, 2021, 10:03am UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/37 "2021-02-19T10:03:53Z")
> [@bbworld1](#): > > I may have missed something but the configuration looks valid to me. Is there some configuration that needs to be done to make it production ready (besides setting the right production urls and values)? In the `docker-compose.yml` file, I see: ```auto - NS_MINIO_URL=minio:9000 - NS_MINIO_RESULTS=http://localhost:8000/api/storage/results # Change - NS_MINIO_ACCESS_KEY=nunaweb - NS_MINIO_SECRET_KEY=supersecurepassword ``` What should these be set to? > [@bbworld1](#): > > This issue has been fixed with the introduction of minio - everything now happens in Celery, and thus this mount is not needed anymore. That being said, we have to integrate the changes before this will work, so we should do that ASAP. I will work on merging the `cicd` branch into `main` today. I will also protect the `main` branch and set the auto-deployment to work on push to `main`. ---
**Author:** ![clydej](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/clydej/32/1098_2.png) [@clydej](https://forum.opencyphal.org/u/clydej)\ **Post date:** [February 19, 2021, 12:52pm UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/38 "2021-02-19T12:52:55Z")
I have rebased `cicd` branch on top of the `main` branch and resolved the file conflicts. It now deploys to production on pushes to `main`. I still need modify the GH Actions build to set the minio password in the `docker-compose.yml` file. We don’t want to set in the file and checked into GH. Could we please test this new deployment again? If minio is working, it should resolve the namespace issues @pavel.kirienko mentioned earlier. ---
**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\ **Post date:** [February 19, 2021, 1:42pm UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/39 "2021-02-19T13:42:38Z")
Okay, yes, the zip is now handled properly but we are still missing `nunavut/serialization`. This is probably related to the application itself, after all. @bbworld1 when you run it locally, does it generate the support library correctly? ---
**Author:** ![bbworld1](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/bbworld1/32/786_2.png) [@bbworld1](https://forum.opencyphal.org/u/bbworld1)\ **Post date:** [February 19, 2021, 6:40pm UTC](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074/40 "2021-02-19T18:40:44Z")
Fixed in [Fix missing support code generation by bbworld1 · Pull Request #13 · OpenCyphal-Garage/nunaweb · GitHub](https://github.com/UAVCAN/nunaweb/pull/13). Apologies for the oversight. ![image](https://forum.opencyphal.org/uploads/default/original/1X/f3550d9da0c4da91f65b1337e1c7ea965c428349.png) [Previous page](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074.md?page=1) [Next page](https://forum.opencyphal.org/t/an-easy-to-use-web-service-for-compiling-dsdl/1074.md?page=3)