Mongo Module¶

A series of functions used for interact directly with mongo client program.

Variables¶

Variable Description

MONGO_CLIENT Path of mongo client.

MONGO_EXTRA_OPTIONS Extra options for mongo client.

MONGO_USER Contains Username to use on connection.

MONGO_PWD Contains password to use on connection.

MONGO_DB Contains database name to use on connection

MONGO_DIR Contains path of directory where found project script.

MONGO_AUTHDB Contains authentication database to use on connection.

MONGO_INITRC Contains path of the file with content to execute before target commands.

Variable	Description
`MONGO_CLIENT`	Path of mongo client.
`MONGO_EXTRA_OPTIONS`	Extra options for mongo client.
`MONGO_USER`	Contains Username to use on connection.
`MONGO_PWD`	Contains password to use on connection.
`MONGO_DB`	Contains database name to use on connection
`MONGO_DIR`	Contains path of directory where found project script.
`MONGO_AUTHDB`	Contains authentication database to use on connection.
`MONGO_INITRC`	Contains path of the file with content to execute before target commands.

Project Folder Structure¶

An initialized project folder is composed by these directories:

creation_scripts: this directory contains initial DDL script for create all collection of the project with master data types. This directory could be used at begin of the project for the first release, in the next release is then used update_scripts directory that contains script for upgrade project and trace changes between releasees.
dbrm-profiles: if profiles are enable contains all configuration files for different environment (dev, test, prod, etc.). This directory could be with a different name and depend of value present of DRM_PROFILES_PATH variable.
indexes: this directory contains files with all indexes of the database. Special index _id could be not present.
update_scripts: this directory is used for store scripts with changes on database between first release and next releases.
schemas: optinal directory that could be contains database schema of the collections for example done by Dia tool.

Under project directory normally is also available dbrm.conf file with main configuration options of the project and dbrm.db sqlite database used by dbrm for the project.

Index Naming Convention¶

Indexes created with create command or downloaded with download command has this naming convention:

  filename = <Collection Name> '.' <Index Name> '.js'

Commands¶

mongo version¶

Show version of mongo module

  $# dbrm mongo version
  Version: 0.1.0

mongo test_connection¶

Test connection to database of the active profile or active configuration.

test_connection options:¶

This options are generic option for handle connection and are avilable on different commands.

-P MONGO_PWD: Override MONGO_PWD variable.
-U MONGO_USER: Override MONGO_USER with username of the connection.
-H MONGO_HOST: Override MONGO_HOST with host of the database.
-D MONGO_DIR: Override MONGO_DIR directory where save/retrieve script/functions, etc.
--database db: Override MONGO_DB variable for database name.
--conn-options opts: Override MONGO_EXTRA_OPTIONS variable for enable extra connection options.
--authdb db: Override MONGO_AUTHDB variable for set authentication database.

  $# dbrm mongo test_connection
  Connected to database1 with user icon correctly.

mongo shell¶

Enter on mongo shell of the active profile or active configuration.

shell options:¶

-P MONGO_PWD: Override MONGO_PWD variable.
-U MONGO_USER: Override MONGO_USER with username of the connection.
-H MONGO_HOST: Override MONGO_HOST with host of the database.
-D MONGO_DIR: Override MONGO_DIR directory where save/retrieve script/functions, etc.
--database db: Override MONGO_DB variable for database name.
--conn-options opts: Override MONGO_EXTRA_OPTIONS variable for enable extra connection options.
--authdb db: Override MONGO_AUTHDB variable for set authentication database.

  $# dbrm mongo shell
  MongoDB shell version v3.4.3
  connecting to: mongodb://192.168.1.200:27018/database1
  MongoDB server version: 3.2.11
  WARNING: shell and server versions do not match
  mongos>

mongo compile¶

Compile files to active database.

From command line is visible result of compilation but a more detailed trace is save on logfile defined on LOGFILE variable (default dbrm.log).

compile options:¶

-P MONGO_PWD: Override MONGO_PWD variable.
-U MONGO_USER: Override MONGO_USER with username of the connection.
-H MONGO_HOST: Override MONGO_HOST with host of the database.
-D MONGO_DIR: Override MONGO_DIR directory where save/retrieve script/functions, etc.
--database db: Override MONGO_DB variable for database name.
--conn-options opts: Override MONGO_EXTRA_OPTIONS variable for enable extra connection options.
--authdb db: Override MONGO_AUTHDB variable for set authentication database.
--file FILE: Compile a particular file. (Use ABS Path or relative path from current dir.)
--indexes: Compile all indexes not already present.
--index INAME: Compila a particular index. This option could be used multiple time. INAME contains name of the collection where drop all indexes of a specific collection or name of the collection with key name separated with a point.

For argument with value if it isn't passed value argument is ignored.

  $#  dbrm mongo compile --file test.js
  (mongo) Start compilation (file test.js): File test.js
  (mongo) End compilation (file test.js, result => 0): File test.js
  Compile operation successfull.

mongo download¶

Download all indexes present on target database.

download options:¶

--indexes: Download all indexes present on database.
--include-id: Download also special index _id that aren't available only for capped collection created with autoIndexId option to false using db.createCollection. On default _id index aren't downloaded.
--index INAME: Index name to download. It is possible download only a single index a time. INAME contains name of the collection for download all indexes of a specific collection or name of the collection with key name separated with a point.

mongo create¶

Create indexes on MongoDB database.

create options:¶

--index INAME: Name of the index to create.. It is possible create only a single index a time. INAME contains name of the collection where create index with index name separated with a point.
--unique: To use with --index option for create an unique index.
--expire-sec SEC: To use with --index option for set expireAfterSeconds option. SEC contains number of seconds.
--opt JSON_CONTENT: Add custom index option as JSON content (without braces). To use with --index option and could be use multiple time.
--key KEY_JSON_CONT: Indicate key fields to use. This option could be used multiple. time for complex keys.
--background: Enable creation of index in background. Default is false.

Example1 - Create an unique index with reverse order¶

Create an unique index for collection test1 on reverse order:

  $# dbrm mongo create --index test1.field1 --unique --key "field1:-1"

Example 2 - Create an expire index¶

Create an expire index for collection test1 of two fields:

  $# dbrm mongo create --index test1.index_m1 --expire-sec 3600 \
    --key "field1:-1" --key "field2:-1"

For sub-document keys add escaped double quote on key option.

mongo drop¶

Drop indexes from database.

drop options:¶

--all-indexes: Drop all indexes present on database (not id).
--index INAME: Index name to drop. This option is not usable multiple time. INAME contains name of the collection where drop all indexes of a specific collection or name of the collection with key name separated with a point.

mongo show¶

Retrieve some informations from database.

show options:¶

--indexes: Show list of indexes present on database.
--collections: Show list of collections present on database.
--all: Show list of all indexes, collections on database.
--collection CNAME: Show detail of a collection. This option could be repeated.
--index INAME: Show detail of a index. This option could be repeated. INAME contains name of the collection for retrieve all indexes of a specific collection or name of the collection with key name separated with a point.
--exclude-id: Exclude special index _id.

API¶

mongo_set_auth_var¶

Set mongo_auth variable with arguments to use with mongo client program. Valorize options like: -u username --password=pwd database.

Currently, authentication with certificate is not yet supported.

Parameters:

$1: (db) Name of the schema to use
$2: (user) User to use on authentication
$3: (pwd) Password to use on authentication.
$4: (host) Optionally host of the database server.
$5: (auth_db) Optionally authentication database to use on authentication.

Returns:

0: always

(Show/Hide)

mongo_file¶

Compile a file and save ouput to input variable.

Parameters:

$1: (var) Name of the variable where is save output command.
$2: (f) File to compile.

Returns:

0: on success
1: on error

(Show/Hide)

mongo_file_initrc¶

Compile a file and save ouput to input variable. File is loaded on memory before execute content, so it isn't ideal for bigfile.

Parameters:

$1: (var) Name of the variable where is save output command.
$2: (f) File to compile.
$3: (initrc) Path of file with commons commands to execute before compile file content. Is an alternative to .mongorc.js file.

Returns:

0: on success
1: on error

(Show/Hide)

mongo_cmd_4var¶

Execute an input statement on configured schema.

Variables Used:

MONGO_CLIENT: Path of mongo client.

Parameters:

$1: (var) Name of the variable where is save output command.
$2: (cmd) Command to execute.
$3: (rm_lf) If string length is not zero than from output command are remove LF.

Returns:

0: on success
1: on error

(Show/Hide)

Commons Mongo API¶

commons_mongo_check_client¶

Check if mongo client program is present on system. If present MONGO_CLIENT variable with abs path is set. Function check if it is set mongo variable:

if it is not set then try to find path through 'which' program
if it is set then check if path is correct and program exists.

Returns:

0: on success
1: on error

(Show/Hide)

commons_mongo_check_vars¶

Check if are present mandatary mongo environment variables:

MONGO_USER
MONGO_PWD
MONGO_DB
MONGO_DIR

Returns:

0: all mandatory variables are present.
1: on error (program is interrupter with exit 1 command)

(Show/Hide)

commons_mongo_check_connection¶

Check connection to database.

Returns:

0: when connection is ok.
1: on error

(Show/Hide)

commons_mongo_shell¶

Enter on command line shell of MongoDB server.

Returns:

0: on success
1: on error

(Show/Hide)

commons_mongo_compile_file¶

Compile a file to current mongodb database.

Parameters:

$1: (file) File to compile.
$2: (msg) message to write on logfile before and after compilation.
$3: (use_initrc) Optional argument to force use of mongo_file_initrc (default) or mongo_file command if equal to 0.

Returns:

0: on success
1: on error

(Show/Hide)

commons_mongo_get_indexes_list¶

Retrieve list of all indexes of current mongodb database and some statistics. This method store result on associative array with name mongo_indexes that contains these columns:

collection: Name of the collection.
key: Name of the index
keys: list of keys of the index separated by comma
n_keys: number of keys of the index
keys_complete: list of the index keys on JSON format
index size: current index size
index options: index options (for example unique, expireAfterSec, etc.)

For identify the size of the array it is needed divide number of records between number of columns (in this case 7) and then iterate for number of elements.

  commons_mongo_get_indexes_list "" "" ""

  local n_indexes=${#mongo_indexes[@]}
  local real_n_indexes=$((n_indexes/7))

  # Retrieve data for every index
  for ((i=0; i<${real_n_indexes}; i++) ; do
      local kcoll=${mongo_indexes[$i, 0]}
      local kname=${mongo_indexes[$i, 1]}
      local n_keys=${mongo_indexes[$i, 3]}
      local keys=${mongo_indexes[$i, 4]}
      local idxsize=${mongo_indexes[$i, 5]}
      local idxopts=${mongo_indexes[$i, 6]}
  done

Parameters:

$1: (single_collection) If present permit to retrieve list of indexes of a particular collection. If not present or equal to empty string than all indexes of database are returned.
$2: (filter_keyname) Optional argument that permit to filter indexes with a particula key name.
$3: (ignore_id_) Optional argument that permit to filter special index _id from list. Possible values are 0 (default) and 1 (ignore index with key name id).

Returns:

0: on success
1: on error

(Show/Hide)

commons_mongo_get_indexes_list () {

  local single_collection="$1"
  local filter_keyname="$2"
  local ignore_id_=${3:-0}
  local pos=0
  local cmd=""

  [[ $DEBUG && $DEBUG == true ]] && echo -en \
    "(commons_mongo_get_indexes_list) args: collection '${single_collection}', filter = '${filter_keyname}', ignore_id_='${ignore_id_}'\n"

  if [ -z "${single_collection}" ] ; then
    cmd="
db.getCollectionNames().forEach(function(n){
  print( JSON.stringify(db[n].getIndexSpecs()));
})"

  else

    cmd="JSON.stringify(db['${single_collection}'].getIndexSpecs())"

  fi


  # Retrieve stats data for index
  commons_mongo_stats "${single_collection}" "index" || return 1

  mongo_cmd_4var "_mongo_ans" "$cmd" || return 1

  # Create associative array for every entry
  # with column:
  # - collection
  # - key name
  # - keys
  # - n_keys
  # - keys_complete
  # - index size
  # - index options

  unset mongo_indexes
  declare -g -A mongo_indexes

  # Reset array and declare it
  mongo_indexes=()

  IFS=$'\n'
  for row in $_mongo_ans ; do

    local n_indexes=$(echo $row | jq 'length')
    local i=0

    for ((i=0; i<${n_indexes};i++)) ; do

      local kname=$(echo $row | jq -r -M .[$i].name )

      if [[ -n "${filter_keyname}" && "${kname}" != ${filter_keyname} ]] ; then
        continue
      fi

      if [[ ${ignore_id_} -eq 1 && "${kname}" == "_id_" ]] ; then
        continue
      fi

      local kcoll=$(echo $row | jq -r -M .[$i].ns | cut -d'.' -f 2)
      local n_keys=$(echo $row | jq ".[$i].key | keys | length")
      local keys=($(echo $row | jq -r -M ".[$i].key | keys | .[]"))
      keys=$(echo ${keys[@]} | tr " " \,)
      local index_size=${mongo_stats[${kcoll}, ${kname}]}

      local keys_complete=$(echo $row | jq -M -c ".[$i].key")

      local key_options=($(echo $row | jq -r -M ".[$i] | keys | .[]"))
      local opts_json="{"

      local opt=""
      for opt in ${key_options[@]} ; do

        local opt_val=""
        case $opt in
          v|key|ns|name)
            ;;
          *)
            opt_val=$(echo $row | jq -M ".[$i].$opt")

            if [ ${#opts_json} -eq 1 ] ; then
              opts_json="${opts_json} \"${opt}\" : ${opt_val}"
            else
              opts_json="${opts_json}, \"${opt}\" : ${opt_val}"
            fi
            ;;
        esac

      done

      if [ ${#opts_json} -eq 1 ] ; then
        opts_json=""
      else
        opts_json="${opts_json} }"
      fi

      # echo -e $(printf '%q' $opts_json) | jq
      [[ $DEBUG && $DEBUG == true ]] && echo -en \
        "(commons_mongo_get_indexes_list) kname = $kname, kcoll =$kcoll, n_keys=$n_keys, keys=$keys_complete.\n"

      mongo_indexes[$pos, 0]=${kcoll}
      mongo_indexes[$pos, 1]=${kname}
      mongo_indexes[$pos, 2]=${keys}
      mongo_indexes[$pos, 3]=${n_keys}
      mongo_indexes[$pos, 4]=${keys_complete}
      mongo_indexes[$pos, 5]=${index_size:-"N.A"}
      mongo_indexes[$pos, 6]=${opts_json}

      let pos++

    done

  done
  unset IFS

  [[ $DEBUG && $DEBUG == true ]] && echo -en \
    "(commons_mongo_get_indexes_list) Found ${#mongo_indexes[@]} ($pos) indexes.\n"

  return 0
}

commons_mongo_stats¶

Retrieve statistics data of collections, indexes or custom stats.

Parameters:

$1: (single_collection) If present permit to retrieve data of a particular collection. If not present or equal to empty string than all collections of database are used for retrieve data.
$2: (target) Identify type of data returned. Possible values are collection, index, custom.
$3: (custom_content) Optional argument that contains jq parse string for retrieve a custom data from json stats of collection. It is used when target contains custom.

Result data are store on mongo_stats array.

If target is equal to collection then is returned an associative array with this columns:

collection: Name of the collection
storageSize: Storage size of the collection
sharded: Identify if collection is sharded (1) or not (0)
primary: If not sharded contains name of the primary node.
count: Number of documents of the collections.
nindexes: Number of indexes of the collections.
totalIndexSize: total size of all indexes of the collection.
capped: If collection is capped (1) or not (0)
avgObjSize: contains average document size
size: size of the collection
sharded nodes: If sharded list of sharded nodes.

For identify the size of the array it is needed divide number of records between number of columns (in this case 11) and then iterate for number of elements.

  commons_mongo_stats "" "collection"

  local n_colls=${#mongo_stats[@]}
  local real_n_colls=$((n_stats/7))

  # Retrieve data for every collection
  for ((i=0; i<${real_n_colls}; i++) ; do
      local coll=${mongo_stats[$i, 0]}
      local storsize=${mongo_stats[$i, 1]}
      local sharded=${mongo_stats[$i, 2]}
      local primary=${mongo_stats[$i, 3]}
      local count=${mongo_stats[$i, 4]}
      local n_idx=${mongo_stats[$i, 5]}
      local totIdx=${mongo_stats[$i, 6]}
      local capped=${mongo_stats[$i, 7]}
      local avgObjSize=${mongo_stats[$i, 8]}
      local size=${mongo_stats[$i, 9]}
      local shardedNodes=${mongo_stats[$i, 10]}
  done

If target is equal to index then is returned an associative array with this columns:

index size: Size of the index.

and where key of the array is composed by collection name and index key name.

  commons_mongo_stats "" "index"

  # Found index size of key "index1" and collection "c1"
  cal indexSize=${mongo_stats["c1", "index1"]}

If target is equal to custom then is returned an associative array with this columns:

collection: Name of the collection
json: Custom json content returned by input js string.

For identify the size of the array it is needed divide number of records between number of columns (in this case 2) and then iterate for number of elements.

  commons_mongo_stats "" "custom" ".wiredTiger.cache"

  local n_colls=${#mongo_stats[@]}
  local real_n_colls=$((n_stats/2))

  # Retrieve data for every collection
  for ((i=0; i<${real_n_colls}; i++) ; do
      local coll=${mongo_stats[$i, 0]}
      local custom_json=${mongo_stats[$i, 1]}
  done

Returns:

0: on success
1: on error

(Show/Hide)

commons_mongo_stats () {

  local scale=1024
  local cmd=""
  local pos=0
  local i=0
  # Contains collection name.
  local single_collection="$1"
  # target values are: collection | index | custom
  local target="$2"
  # Custom content permit to customize filter for
  # jq parser and return JSON to process.
  local custom_content="$3"

  [[ $DEBUG && $DEBUG == true ]] && echo -en \
    "(commons_mongo_mongo_stats) args: $@.\n"

  if [ -z "${single_collection}" ] ; then
    cmd="
db.getCollectionNames().forEach(function(n) {
  print( JSON.stringify(db[n].stats({scale: ${scale}})) );
})"
  else
    cmd="JSON.stringify(db['${single_collection}'].stats({scale: ${scale}}))"
  fi

  mongo_cmd_4var "_mongo_stats" "$cmd" || return 1

  unset mongo_stats
  declare -g -A mongo_stats
  mongo_stats=()

  IFS=$'\n'
  for row in $_mongo_stats ; do

    local row_coll=$(echo $row | jq -r -M .ns | cut -d'.' -f 2)

    case "${target}" in
      collection)

        # Create associative array with columns (number index):
        # - collection
        # - storageSize
        # - sharded
        # - primary
        # - count
        # - nindexes
        # - totalIndexSize
        # - capped
        # - average object size
        # - size
        # - sharded nodes
        local storageSize=""
        local sharded=""
        local primary=""
        local count=""
        local nindexes=""
        local totalIndexSize=""
        local capped=""
        local avgObjSize=""
        local size=""
        local shardedNodes=""

        storageSize=$(echo $row | jq -r -M .storageSize)
        sharded=$(echo $row | jq -r -M .sharded)

        count=$(echo $row | jq -r -M .count)
        nindexes=$(echo $row | jq -r -M .nindexes)
        totalIndexSize=$(echo $row | jq -r -M .totalIndexSize)
        capped=$(echo $row | jq -r -M .capped)
        avgObjSize=$(echo $row | jq -r -M .avgObjSize)
        size=$(echo $row | jq -r -M .size)

        mongo_stats[$pos, 0]=${row_coll}
        mongo_stats[$pos, 1]=${storageSize}

        if [ "${sharded}" = "false" ] ; then
          mongo_stats[$pos, 2]=0
          primary=$(echo $row | jq -r -M .primary)
          shardedNodes="-"
        else
          mongo_stats[$pos, 2]=1
          primary="-"
          shardedNodes=($(echo $row | jq -r -M ".shards | keys | .[]"))
          shardedNodes=$(echo ${shardedNodes[@]} | tr " " \,)
        fi
        mongo_stats[$pos, 3]=${primary}
        mongo_stats[$pos, 4]=${count}
        mongo_stats[$pos, 5]=${nindexes}
        mongo_stats[$pos, 6]=${totalIndexSize}
        if [ "${capped}" = "false" ] ; then
          mongo_stats[$pos, 7]=0
        else
          mongo_stats[$pos, 7]=1
        fi
        mongo_stats[$pos, 8]=${avgObjSize}
        mongo_stats[$pos, 9]=${size}
        mongo_stats[$pos, 10]=${shardedNodes}
        ;;

      index)

        # Create associative array with columns (key is collection, index name):
        # - index size
        local indexSize=""
        local nkeys=$(echo $row | jq -r -M ".indexSizes | length")

        if [ ${nkeys} -gt 0 ] ; then

          local keys=($(echo $row | jq -r -M ".indexSizes | keys | .[]"))

          for key in ${keys[@]} ; do

            indexSizes=$(echo $row | jq -r -M ".indexSizes.${key}")
            mongo_stats[${row_coll}, ${key}]=${indexSizes}

          done # end for key

        fi

        ;;

      custom)
        # Create associative array with columns for collection (number index):
        # - collection
        # - custom json
        local custom_json=$(echo $row | jq -M "${custom_content}")

        mongo_stats[$pos, 0]=${row_coll}
        mongo_stats[$pos, 1]=${custom_json}

        ;;
    esac

    let pos++

  done # end for
  unset IFS

  [[ $DEBUG && $DEBUG == true ]] && echo -en \
    "(commons_mongo_stats) Found ${#mongo_stats[@]} ($pos) data.\n"

  return 0
}

commons_mongo_create_index_file¶

Create index file for compilation under $MONGO_DIR/indexes directory. This function create only the file without execute it on database.

Parameters:

$1: (coll) name of the collection where create index.
$2: (name) Name of the index to create.
$3: (keys) list of keys as json content to use for create index.
$4: (opts) list of key options on json format to use on creation script.

Returns:

0: on success
1: on error

  # Create index for collection c1 for fiedl f1 with option unique.
  commons_mongo_create_file "c1" "indexf1" "{f1:-1}" "{\"unique\":true}"

(Show/Hide)

commons_mongo_download_index¶

Download and index configuration and create a creation script under `$MONGO_DIR/indexes directory.

Parameters:

$1: (coll) name of the collection where download index data.
$2: (kname) Name of the index to download.
$3: (arr_pos) Optional parameter that contains position of the index to download when there is already an valorized array mongo_indexes previously initialized. If not present function download indipendently index data through commons_mongo_get_indexes_list function.

Returns:

0: on success
1: on error

(Show/Hide)

commons_mongo_download_all_indexes¶

Download all indexes to MONGO_DIR/indexes directory.

Parameters:

$1: (collection) download all indexes of input collection. Empty string means all indexes.
$2: (include_id_) Optional parameter that could be set to 0 (default value) when special index _id are ignored from download or set 1 when on download are created also files related with special index _id.

Returns:

0: on success
1: on error

(Show/Hide)

commons_mongo_drop_index¶

Drop a index from database if exists.

Parameters:

$1: (coll) name of the collection where drop index.
$2: (kname) Name of the index to drop. If not present then all indexes of the collection are dropped. Excluded special index that can be removed only for capped collection.

Returns:

0: on success
1: on error

(Show/Hide)

commons_mongo_compile_idx¶

Compile file related with index of a collection..

Parameters:

$1: (f) path of the file to compile
$2: (msg) message to insert on logging file relative to input file
$3: (force) if index is present and force is equal to 1, then index is dropped and added again. NOT IMPLEMENTED.

Returns:

0: on success
1: on error

(Show/Hide)

commons_mongo_compile_all_from_dir¶

Compile all files from input directory with .js extension.

Parameters:

$1: (directory) Directory where there are files to compile.
$2: (msg_head) Title message insert on logfile before compile files.
$3: (msg) message insert on logfile before compile files.
$4: (type) Identify type of directory: idx or empty. Optional.
$5: (closure) custom parameter with a value relative to type.

Returns:

0: on success
1: on error

(Show/Hide)

commons_mongo_compile_all_idxs¶

Compile all files under $MONGO_DIR/indexes directory.

Parameters:

$1: (msg) message to insert on logging file relative to input file.
$2: (force) if equals to 1, force compilation of all indexes also if already present.

Returns:

0: on success
1: on error

(Show/Hide)