Recently I had an interesting problem. The product I was working on needed to create a secret from a file. Now on one hand this is an easy thing as you can just have a job run within an image

kubectl create secret generic mysecret --from-file=./file.txt

Ah… However Kubernetes command line client is only compatible within one minor version of the Kubernetes api server. So if you want to support all major version for Redhat Openshift currently supported you have to support Kubernetes 1.11 to 1.21. So to get around this problem you have to curl against the kubapi server directly. Here is a sample script I created to demonstrate this method.

apiVersion: batch/v1
kind: Job
metadata:
  name: readfiletosecret
  description: "Example of reading a file to a kubernetes secret."
spec:
  template:
    spec:
      serviceAccountName: account-with-secret-create-priv
      volumes:
      - name: local
        emptyDir: {}
      initContainers:
      - name: get-file
        image: registry.access.redhat.com/ubi8/ubi-minimal:latest
        command:
        - "/bin/sh"
        - "-c"
        env:
        - name: UPLOAD_FILE_PATH
          value: "/root/buildinfo/content_manifests/ubi8-minimal-container*.json"
        args:
        - |
          cat $UPLOAD_FILE_PATH
          cp -vf $UPLOAD_FILE_PATH /work/
        volumeMounts:
        - name: local
          mountPath: /work
      containers:
      - name: create-secret
        image: registry.access.redhat.com/ubi8/ubi-minimal:latest
        command:
        - "/bin/sh"
        - "-c"
        args:
        - |
          ls /work/
          i#
          export CONTENT=$(cat /work/* | base64 )
          echo $CONTENT
          #Set auth info
          export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
          export NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
          export TOKEN=$(cat ${SERVICEACCOUNT}/token)
          export CACERT=${SERVICEACCOUNT}/ca.crt
          export APISERVER="https://kubernetes.default.svc"
          # Explore the API with TOKEN
          curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' -d @-  ${APISERVER}/api/v1/namespaces/$NAMESPACE/secrets <<EOF
          {
            "kind": "Secret",
            "apiVersion": "v1",
            "metadata": {
              "name": "example"
            },
            "data": {
              "file": "$CONTENT"
            }
          }
          EOF
          rm /work/*
        volumeMounts:
        - name: local
          mountPath: /work
      restartPolicy: Never
  backoffLimit: 4

Iteration 0 is often used to create a product backlog or setup technical foundation (code repos, build pipelines, etc..). I frankly thought the concept was odd. In Agile you work to dates not scope. So to have an iteration that defines scope is odd. Thus I have a different take on it.

Iteration 0 is when you “start to know what you don’t” and with each subsequent iteration you learn more and the plan changes; So with that said it is time to prime the product backlog. Let’s start with defining goals.

Goal #1: Port Lucene from Java to Rust

While it seems straight forward we need to understand what Java parts are in the Lucene project. So let’s start start with a…

git clone https://github.com/apache/lucene.git
cd lucene
ls                                            
LICENSE		build.gradle	dev-docs	gradle		gradlew.bat	lucene		versions.lock
README.md	buildSrc	dev-tools	gradlew		help		settings.gradle	versions.props

OK, So right off the bat I can see it is a Gradle project. Gradle supports multiple languages out of the box and supports plugins to support others like Rust. Here we have our first decision… Do we port the Lucene library code or do we port the project? Let’s revisit that later after we are done exploring.

The interesting thing is there are a good number of files that are very specific to how Apache runs their projects and performs releases. For instance I haven’t seen an RDF document in probably 10 years yet here is a DOAP document defining the project and all the releases per Apache standards. The two items which I should focus on is the Lucene directory with all the java files and dev-tools/scripts/smokeTestRelease.py. The later will be useful for the next goal.

Goal #2: It should pass existing test scripts

So I think this is something which will set the project apart from other ports. Providing a way to reuse existing test scripts will ensure compatibility over time. However doing so will require the ability to do 2 things.

Leverage JNI from RUST

It has been a while since I’ve used JNI so this will be a good refresher. Based on existing documentation it seems this is done all the time for Android but still an experiment is required to ensure there are not any unforeseen gotchas.

The ability to sync Tests with the parent project

This is going to be a hard one. Reading through dev-tools/scripts/smokeTestRelease.py the Release verification is more than just Unit Tests. It also verifies digest mismatch, documentation, and missing metadata. Not all of these verifications will be applicable. For instance verifying the jar “Implementation-Vendor”metadata would not apply. So valid parts of this script will need to be ported and maintained.

At first glance majority of the verification is in the form of unit tests. In fact it looks like roughly 33% of all the Java source files are unit tests. That being said the Gradle build handles preparing data which may be used in those tests.

find lucene/ |grep ".java" |wc -l
5505
find lucene/ |grep ".java" | grep test | wc -l
1844

So we probably need a mechanism to

• pull the latest project from Lucene
• Clear out the non-test related Java source files
• Update the build scripts to leverage an external JAR
• Run the tests

Of course this means the code for the port needs to be managed separately from Lucene. This is turning into more of a complex project than I expected. Going to need to spend some time understanding a good project setup which will allow this. However it is currently the end of this iteration so that will have to wait for the next one.

It goes without saying that 2020 was a tough year. While 2021 is getting better we are still not out of the woods yet. Sometimes the best way to get through it is to have a project that isn’t related to work but helps exercise the mind. Now I am not actually building a boat. My wife would kill me. However I am starting a project of similar ambition.

While granted I am a manager I still like to keep my development skills sharp. This is why I jump in to write code that helps my team reach their goals. This can range from everything from GO to Python. So for a personal project I don’t want to program in any of those.

Thus I am going to take a few open source projects and port them to Rust. I am gong document my experience and what works/doesn’t. Now I intent to do something a bit different for this port. I am going to use JNI to ensure the port is 100% compatible with the old one.

Choosing the right project

It goes without saying that there are plenty of open source projects. So here is the criteria I am using for selecting the right project to port.

• It should be an established project which is not radically changing. This will make maintenance of the port easer over time.
• It should have existing ports and the community should be open to new ports.
• It should be large but not so large that it can never be completed.
• It should be a project I am familiar with.

Based on this criteria I have decided to port the Lucene project. I have worked with Lucene on multiple projects back in the day. It will be good to revisit it and understand better how it works under the hood.