Implemented options_from form value for select and radio, allowing dynamic options

doc/MANUAL.md

@@ -39,6 +39,7 @@ This is the manual for version %%VERSION%%.
     - [Text](#field_types_text)
     - [Password](#field_types_password)
     - [File](#field_types_file)
+1. [Dynamic forms](#dynamic_forms)
 1. [Output](#output)
     - [Output types](#output_types)
     - [Exit codes](#output_exitcodes)
@@ -882,6 +883,10 @@ The `radio` field type lets the user pick one option from a list of options.
 The `radio` field type supports the following additional options:
 
 - **`options`**: The options available to the user. (list of lists, **required**)
+- **`options_from`**: A list of available options from which the user can
+  choose, read from an external file or executable. 
+
+Either `options` or `options_from` may be given, but not both.
 
 For example:
 
@@ -936,14 +941,18 @@ The `select` field type supports the following additional options:
 - **`options`**: A list of available options from which the user can choose.
   Each item in the list is itself a list of two values: the value and the
   title.
+- **`options_from`**: A list of available options from which the user can
+  choose, read from an external file or executable. 
+
+Either `options` or `options_from` may be given, but not both.
 
 For example
 
     ...
     "fields": [
         {
-          "name": "source_sql",
+          "name": "target_db",
-          "title": "Load which kind of database?",
+          "title": "Load CSV into which database?",
           "type": "select",
           "options": [
             ["empty", "Empty database"],
@@ -954,6 +963,18 @@ For example
     ]
     ...
 
+Dynamically read select options from a script:
+
+    ...
+    "fields": [
+        {
+          "name": "target_db",
+          "title": "Load CSV into which database?",
+          "type": "select",
+          "options_from": "form_loadcsv_target_db.sh"
+        }
+    ]
+    ...
 
 ### <a name="field_types_text">Text</a>
 
@@ -1022,6 +1043,79 @@ For example:
 
 
 
+## <a name="dynamic_forms">Dynamic Forms</a>
+
+Often it won't be enough to have staticly defined fields in forms. For
+example, you may  want the user to be able to select from a dynamically
+created list of files or databases.
+
+Scriptform offers a way to fill in parts of a form definition file dynamically
+from extern files or scripts. This is done at runtime when showing or
+validating the form, so no restarting of scriptform is required.
+
+When it's possible to dynamically load parts of a form, the form definition
+haev a `_from` option. For example, normally you'd use the `options` element
+for a `select` field in  a form, like so:
+
+    ...
+    "fields": [
+        {
+          "name": "target_db",
+          "title": "Load CSV into which database?",
+          "type": "select",
+          "options": [
+            ["empty", "Empty database"],
+            ["dev", "Development test database"],
+            ["ua", "Acceptance database"]
+          ]
+        }
+    ]
+    ...
+
+Scriptform also understands the `options_from` field, which replaces the
+`options` field. The value should point to a normal or executable file.
+Depending on that, the file is read or executed and its output interpreted as
+JSON. The form then uses that JSON as a replacement for the `options` field.
+
+For example:
+
+    ...
+    "fields": [
+        {
+          "name": "target_db",
+          "title": "Load CSV into which database?",
+          "type": "select",
+          "options_from": "form_loadcsv_target_db.sh"
+        }
+    ]
+    ...
+
+The script `form_loadcsv_target_db.sh` could look like:
+
+    #!/bin/sh
+    
+    cat << END_TEXT
+    [
+        ["test", "Test DB"],
+        ["acc", "Acc DB"],
+        ["prod", "Prod DB"]
+    ]
+    END_TEXT
+
+Notes:
+
+* The executable bit must be set in order for scriptform to execute the file.
+  Otherwise, the contents of the file is read.
+* Executable scripts are *always* executed as the user scriptform runs at!
+  While its not possible for the user to inject anything into the script, you
+  should still be careful with what the scripts do.
+
+### Supported dynamic form parts
+
+Currently the following parts of form definitions support dynamic form parts:
+
+* `fields`: `type` = `select`.
+
 
 ## <a name="output">Output</a>
 

examples/dynamic_forms/README.md

@@ -0,0 +1,5 @@
+ScriptForm dynamic forms
+========================
+
+This example shows how to create dynamic forms where parts of the form are
+generated by external scripts.

examples/dynamic_forms/dynamic_forms.json

@@ -0,0 +1,25 @@
+{
+    "title": "Dynamic forms",
+    "forms": [
+        {
+            "name": "import",
+            "title": "Import CSV data",
+            "description": "Import CSV into a database",
+            "submit_title": "Import",
+            "script": "job_import.sh",
+            "fields": [
+                {
+                    "name": "target_db",
+                    "title": "Database to import to",
+                    "type": "radio",
+                    "options_from": "form_import_target_dbs.sh"
+                },
+                {
+                    "name": "sql_file",
+                    "title": "SQL file",
+                    "type": "file"
+                }
+            ]
+        }
+    ]
+}

examples/dynamic_forms/form_import_target_dbs.sh

@@ -0,0 +1,9 @@
+#!/bin/sh
+
+cat << END_TEXT
+[
+    ["test", "Test DB"],
+    ["acc", "Acc DB"],
+    ["prod", "Prod DB"]
+]
+END_TEXT

examples/dynamic_forms/job_import.sh

@@ -0,0 +1,10 @@
+#!/bin/sh
+
+
+MYSQL_DEFAULTS_FILE="my.cnf"
+MYSQL="mysql --defaults-file=$MYSQL_DEFAULTS_FILE"
+
+echo "This is what would be executed if this wasn't a fake script:"
+echo 
+echo "echo 'DROP DATABASE $target_db' | $MYSQL"
+echo "$MYSQL ${target_db} < ${sql_file}"

src/formdefinition.py

@@ -6,6 +6,8 @@ validation of the form values.
 import os
 import datetime
 
+import runscript
+
 
 class ValidationError(Exception):
     """Default exception for Validation errors"""
@@ -190,8 +192,15 @@ class FormDefinition(object):
         """
         Validate a form field of type 'radio'.
         """
+        if 'options_from' in field_def:
+            # Dynamic options from file
+            active_options = runscript.from_file(field_def["options_from"])
+        else:
+            # Static options defined in form definition
+            active_options = field_def['options']
+
         value = form_values[field_def['name']]
-        if value not in [o[0] for o in field_def['options']]:
+        if value not in [o[0] for o in active_options]:
             raise ValidationError(
                 "Invalid value for radio button: {0}".format(value))
         return value
@@ -200,8 +209,15 @@ class FormDefinition(object):
         """
         Validate a form field of type 'select'.
         """
+        if 'options_from' in field_def:
+            # Dynamic options from file
+            active_options = runscript.from_file(field_def["options_from"])
+        else:
+            # Static options defined in form definition
+            active_options = field_def['options']
+
         value = form_values[field_def['name']]
-        if value not in [o[0] for o in field_def['options']]:
+        if value not in [o[0] for o in active_options]:
             raise ValidationError(
                 "Invalid value for dropdown: {0}".format(value))
         return value

src/runscript.py

@@ -8,6 +8,40 @@ import os
 import pwd
 import grp
 import subprocess
+import json
+
+
+def from_file(fname):
+    """
+    Read or execute `fname` and decode its contents as JSON. Used for reading
+    parts of forms from external files or scripts.
+    """
+    log = logging.getLogger(__name__)
+
+    if not fname.startswith('/'):
+        path = os.path.join(os.path.realpath(os.curdir), fname)
+    else:
+        path = fname
+
+    if os.access(path, os.X_OK):
+        # Executable. Run and grab output
+        log.debug("Executing %s", path)
+        proc = subprocess.Popen(path,
+                                shell=True,
+                                stdout=subprocess.PIPE,
+                                stderr=subprocess.PIPE,
+                                close_fds=True)
+        stdout, stderr = proc.communicate(input)
+        if proc.returncode != 0:
+            log.error("%s returned non-zero exit code %s", path, proc.returncode)
+            log.error(stderr)
+            raise subprocess.CalledProcessError(proc.returncode, path, stderr)
+        out = stdout
+    else:
+        # Normal file
+        with open(path, 'r') as filehandle:
+            out = filehandle.read()
+    return json.loads(out)
 
 
 def run_as(uid, gid, groups):

src/webapp.py

@@ -309,12 +309,22 @@ class ScriptFormWebApp(RequestHandler):
                 params['cols'] = field.get('cols', '')
 
             if field['type'] == 'radio':
+                if 'options_from' in field:
+                    fname = field['options_from']
+                    options = runscript.from_file(fname)
+                else:
+                    options = field['options']
+
                 if not form_values.get(field['name'], None):
-                    params['value'] = field['options'][0][0]
+                    # Set default value
-                params['options'] = field['options']
+                    params['value'] = options[0][0]
 
             if field['type'] in ('radio', 'select'):
-                params['options'] = field['options']
+                if 'options_from' in field:
+                    fname = field['options_from']
+                    params['options'] = runscript.from_file(fname)
+                else:
+                    params['options'] = field['options']
 
             if field['type'] == 'checkbox':
                 # Set default value from field definition