.. This work is licensed under a Creative Commons Attribution 4.0 .. International License. http://creativecommons.org/licenses/by/4.0 .. Copyright 2017 AT&T Intellectual Property. All rights reserved. Setting Up ========== ONAP documentation is stored in git repositories, changes are managed with gerrit reviews, and published documents generated when there is a change in any source used to build the documentation. Authors create source for documents in reStructured Text (RST) that is rendered to HTML and published on Readthedocs.io. The developer Wiki or other web sites can reference these rendered documents directly allowing projects to easily maintain current release documentation. Some initial set up is required to connect a project with the master document structure and enable automated publishing of changes as summarized in the following diagram and description below below. .. seqdiag:: :height: 700 :width: 1000 seqdiag { DA [label = "Doc Project\nAuthor/Committer", color=lightblue]; DR [label = "Doc Gerrit Repo" , color=pink]; PR [label = "Other Project\nGerrit Repo", color=pink ]; PA [label = "Other Project\nAuthor/Committer", color=lightblue]; PA -> DR [label = "Add project repo as\ngit submodule" ]; DR -> DA [label = "Review & Plan to\nIntegrate Content with\nTocTree Structure" ]; DR <-- DA [label = "Vote +2/Merge" ]; PA <-- DR [label = "Merge Notification" ]; PA -> PR [label = "Create in project repo\ntop level directory and index.rst" ]; PR -> DA [label = "Add as Reviewer" ]; PR <-- DA [label = "Approve and Integrate" ]; PA <-- PR [label = "Merge" ]; } Setup project repositories -------------------------- These steps are performed for each project repository that provides documentation. 1. Set two variables that will be used in the subsequent steps. Set *reponame* to the project repository you are setting up just as it appears in the **Project Name** column of the Gerrit projects page. Set *lfid* to your Linux Foundation identity that you use to login to gerrit or for git clone requests over ssh. .. code-block:: bash reponame= lfid= 2. Add a directory in the doc project where your project will be included as a submodule and at least one reference from the doc project to the documentation index in your repository. The following sequence will do this over ssh. Please note that the reference to your project in *repolist.rst* should be considered temporary and removed when you reference it from more appropriate place. .. caution:: If your access network restricts ssh, you will need to use equivalent git commands and HTTP Passwords as described `here `_. .. caution:: Don't replace ../ in *git submodule add* with any relative path on your local file system. It refers to the location of your repository on the server. .. code-block:: bash git clone ssh://$lfid@gerrit.onap.org:29418/doc cd doc mkdir -p `dirname docs/submodules/$reponame` git submodule add ../$reponame docs/submodules/$reponame.git git submodule init docs/submodules/$reponame.git git submodule update docs/submodules/$reponame.git echo " $reponame <../submodules/$reponame.git/docs/index>" >> docs/release/repolist.rst git add . git commit -s git review .. caution:: Wait for the above change to be merged before any merge to the project repository that you have just added as a submodule. If the project repository added as submodule changes before the doc project merge, git may not automatically update the submodule reference on changes and/or the verify job will fail in the step below. 3. Create a docs directory in your repository with an index.rst file. The following sequence will complete the minimum required over ssh. As you have time to convert or add new content you can update the index and add files under the docs folder. .. hint:: If you have additional content, you can include it by editing the index.rst file and/or adding other files before the git commit. See `Templates and Examples`_ below and :ref:`converting-to-rst` for more information. .. code-block:: bash git clone ssh://$lfid@gerrit.onap.org:29418/$reponame cd $reponame mkdir docs echo ".. This work is licensed under a Creative Commons Attribution 4.0 International License. TODO Add files to toctree and delete this header ------------------------------------------------ .. toctree:: :maxdepth: 1 " > docs/index.rst git add . git commit -s git review The diagram below illustrates what is accomplished in the setup steps above from the perspective of a file structure created for a local test, a jenkins verify job, and/or published release documentation including: - ONAP gerrit project repositories, - doc project repository master document index.rst, templates, configuration, and other documents - submodules directory where other project repositories and directories/files are referenced - file structure: directories (ellipses), files(boxes) - references: directory/files (solid edges), git submodule (dotted edges), sphinx toctree (dashed edges) .. graphviz:: digraph docstructure { size="8,12"; node [fontname = "helvetica"]; // Align gerrit repos and docs directories {rank=same doc aaf aai reponame repoelipse vnfsdk vvp} {rank=same confpy release templates masterindex submodules otherdocdocumentelipse} {rank=same releasedocumentindex releaserepolist} //Illustrate Gerrit Repos and provide URL/Link for complete repo list gerrit [label="gerrit.onap.org/r", href="https://gerrit.onap.org/r/#/admin/projects/" ]; doc [href="https://gerrit.onap.org/r/gitweb?p=doc.git;a=tree"]; gerrit -> doc; gerrit -> aaf; gerrit -> aai; gerrit -> reponame; gerrit -> repoelipse; repoelipse [label=". . . ."]; gerrit -> vnfsdk; gerrit -> vvp; //Show example of local reponame instance of component info reponame -> reponamedocsdir; reponamesm -> reponamedocsdir; reponamedocsdir [label="docs"]; reponamedocsdir -> repnamedocsdirindex; repnamedocsdirindex [label="index.rst", shape=box]; //Show detail structure of a portion of doc/docs doc -> docs; docs -> confpy; confpy [label="conf.py",shape=box]; docs -> masterindex; masterindex [label="Master\nindex.rst", shape=box]; docs -> release; docs -> templates; docs -> otherdocdocumentelipse; otherdocdocumentelipse [label="...other\ndocuments"]; docs -> submodules masterindex -> releasedocumentindex [style=dashed, label="sphinx\ntoctree\nreference"]; //Show submodule linkage to docs directory submodules -> reponamesm [style=dotted,label="git\nsubmodule\nreference"]; reponamesm [label="reponame.git"]; //Example Release document index that references component info provided in other project repo release -> releasedocumentindex; releasedocumentindex [label="index.rst", shape=box]; releasedocumentindex -> releaserepolist [style=dashed, label="sphinx\ntoctree\nreference"]; releaserepolist [label="repolist.rst", shape=box]; release -> releaserepolist; releaserepolist -> repnamedocsdirindex [style=dashed, label="sphinx\ntoctree\nreference"]; } Branches in the DOC Project --------------------------- The DOC project 'master' branch aggregates the 'latest' content from all ONAP project repositories contributing documentation into a single tree file structure as described in the previous section. This branch is continuously integrated and deployed at Read The Docs as the 'latest' ONAP Documentation by: * Jenkins doc-verify-rtd and doc-merge-rtd jobs triggered whenever patches on contributing repositories contain rst files at or below a top level 'docs' folder. * Subscription in the DOC project to changes in submodule repositories. These changes appear in the DOC project as commits with title 'Updated git submodules' when a change to a contributing project repository is merged. No DOC project code review occurs, only a submodule repository commit hash is updated to track the head of each contributing master branch. For each ONAP named release the DOC project creates a branch with the release name. The timing of the release branch is determined by work needed in the DOC project to prepare the release branch and the amount of change unrelated to the release in the master branch. For example contributing projects that create named release branches early to begin work on the next release and/or contributing projects to the master that are not yet part of the named release would result in an earlier named release branch to cleanly separate work to stabilize a release from other changes in the master branch. A named release branch is integrated and deployed at Read The Docs as the 'named release' by aggregating content from contributing project repositories. A contributing project repository can choose one of the following for the 'named release' branch: * Remove the contributing project repository submodule and RST references when not part of the named release. * Provide a commit hash or tag for the contributing project master branch to be used for the life of the release branch or until a request is submitted to change the commit hash or tag. * Provide the commit hash for the head of a named release branch created in the contributing project repository. This option may be appropriate if frequent changes are expected over the life of the named release and work the same way as the continuous integration and deployment described for the master branch. The decision on option for each contributing project repository can be made or changed before the final release is approved. The amount of change and expected differences between master and a named release branch for each repository should drive the choice of option and timing. About GIT branches ------------------ GIT is a powerful tool allowing many actions, but without respecting some rules the GIT structure can be quickly hard to maintain. Here are some conventions about GIT branches: - ALWAYS create a local branch to edit or create any file. This local branch will be considered as a topic in Gerrit and allow contributors to work at the same time on the same project. - 1 feature = 1 branch. In the case of documentation, a new chapter or page about a new code feature can be considered as a 'doc feature' - 1 bug = 1 branch. In the case of documentation, a correction on an existing sentence can be considered as a 'doc bug' - the master branch is considered as "unstable", containing new features that will converge to a stable situation for the release date. The day of the release, the repository owner will create a new branch to fix the code and documentation. This will represent the 'stable' code of the release. In this context: - NEVER push a new feature on a stable branch - Only bug correction are authorized on a stable branch using cherry pick method .. image:: git_branches.png Creating Restructured Text ========================== ReStructuredText markup conventions ----------------------------------- For detailed information on ReStructuredText and how to best use the format, see: - `ReStructured Text Primer `_ - `ReStructured Text Quick Reference `_ Templates and Examples ---------------------- Templates are available that capture the kinds of information useful for different types of projects and provide some examples of restructured text. We organize templates in the following way to: - help authors understand relationships between documents - keep the user audience context in mind when writing and - tailor sections for different kinds of projects. **Sections** Represent a certain type of content. A section is **provided** in an project repository, to describe something about the characteristics, use, capability, etc. of things in that repository. A section may also be **referenced** from other sections and in other repositories. For example, an API specification provided in a project repository might be referenced to in a Platform API Reference Guide. The notes in the beginning of each section template provide additional detail about what is typically covered and where there may be references to the section. **Collections** Are a set of sections that are typically provided for a particular type of project, repository, guide, reference manual, etc. For example, a collection for a platform component, an SDK, etc. You can: browse the template *collections* and *sections* below; show source to look at the Restructured Text and Sphinx directives used. Sections ++++++++ Section examples are available here: :ref:`Templates` Collections +++++++++++ In addition to these simple templates and examples there are many open source projects (e.g. Open Daylight, Open Stack) that are using Sphinx and Readthedocs where you may find examples to start with. Working with project teams we will continue to enhance templates here and capture frequently asked questions on the developer wiki question topic `documentation `_. Each project should: - decide what is relevant content - determine the best way to create/maintain it in the CI/CD process and - work with the documentation team to reference content from the master index and guides. Consider options including filling in a template, identifying existing content that can be used as is or easily converted, and use of Sphinx directives/extensions to automatically generate restructured text from other source you already have. Collection examples are available here: :ref:`Templates` Links and References -------------------- It's pretty common to want to reference another location in the ONAP documentation and it's pretty easy to do with reStructuredText. This is a quick primer, more information is in the `Sphinx section on Cross-referencing arbitrary locations `_. Within a single document, you can reference another section simply by:: This is a reference to `The title of a section`_ Assuming that somewhere else in the same file there a is a section title something like:: The title of a section ^^^^^^^^^^^^^^^^^^^^^^ It's typically better to use ``:ref:`` syntax and labels to provide links as they work across files and are resilient to sections being renamed. First, you need to create a label something like:: .. _a-label: The title of a section ^^^^^^^^^^^^^^^^^^^^^^ .. note:: The underscore (_) before the label is required. Then you can reference the section anywhere by simply doing:: This is a reference to :ref:`a-label` or:: This is a reference to :ref:`a section I really liked ` .. note:: When using ``:ref:``-style links, you don't need a trailing underscore (_). Because the labels have to be unique, it usually makes sense to prefix the labels with the project name to help share the label space, e.g., ``sfc-user-guide`` instead of just ``user-guide``. Index File ---------- The index file must relatively reference your other rst files in that directory. Here is an example index.rst : .. code-block:: bash ******************* Documentation Title ******************* .. toctree:: :numbered: :maxdepth: 2 documentation-example Source Files ------------ Document source files have to be written in reStructuredText format (rst). Each file would be built as an html page. Here is an example source rst file : .. code-block:: bash ============= Chapter Title ============= Section Title ============= Subsection Title ---------------- Hello! Writing RST Markdown -------------------- See http://sphinx-doc.org/rest.html . **Hint:** You can add html content that only appears in html output by using the 'only' directive with build type ('html' and 'singlehtml') for an ONAP document. But, this is not encouraged. .. code-block:: bash .. only:: html This line will be shown only in html version. Creating Indices ---------------- Building an index for your Sphinx project is relatively simple. First, tell Sphinx that you want it to build an index by adding something like this after your TOC tree: .. code-block:: rst Indices and Search ================== * :ref:`genindex` * :ref:`search` **Hint:** Note that search was included here. It works out of the box with any Sphinx project, so you don't need to do anything except include a reference to it in your :code:`index.rst` file. Now, to generate a index entry in your RST, do one of the following: .. code-block:: rst Some content that requires an :index:`index`. or .. code-block:: rst .. index:: single: myterm Some header containing myterm ============================= In the second case, Sphinx will create a link in the index to the paragraph that follows the index entry declaration. When your project is built, Sphinx will generate an index page populated with the entries you created in the source RST. These are simple cases with simple options. For more information about indexing with Sphinx, please see the `official Sphinx documentation `_. Jenkins Jobs ------------ Verify Job ++++++++++ The verify job name is **doc-{stream}-verify-rtd** Proposed changes in files in any repository with top level docs folder in the repository and RST files in below this folder will be verified by this job as part of a gerrit code review. .. Important:: The contributing author and every reviewer on a gerrit code review should always review the Jenkins log before approving and merging a change. The log review should include: * Using a browser or other editor to search for a pattern in the *console log* that matches files in the patch set. This will quickly identify errors and warnings that are related to the patch set and repository being changed. * Using a browser to click on the *html* folder included in the log and preview how the proposed changes will look when published at Read The Docs. Small changes can be easily made in the patch set. Merge Job +++++++++ The merge job name is **doc-{stream}-merge-rtd**. When a committer merges a patch that includes files matching the path described above, the doc project merge job will trigger an update at readthedocs. There may be some delay after the merge job completes until new version appears at Read The Docs. Testing ======= One RST File ------------ It is recommended that all rst content is validated by `doc8 `_ standards. To validate your rst files using doc8, install doc8. .. code-block:: bash sudo pip install doc8 doc8 can now be used to check the rst files. Execute as, .. code-block:: bash doc8 --ignore D000,D001 One Project ----------- To test how the documentation renders in HTML, follow these steps: Install `virtual environment `_ & create one. .. code-block:: bash sudo pip install virtualenv virtualenv onap_docs Activate `onap_docs` virtual environment. .. code-block:: bash source onap_docs/bin/activate .. note:: Virtual environment activation has to be performed before attempting to build documentation. Otherwise, tools necessary for the process might not be available. Download a project repository. .. code-block:: bash git clone http://gerrit.onap.org/r/ Download the doc repository. .. code-block:: bash git clone http://gerrit.onap.org/r/doc Change directory to doc & install requirements. .. code-block:: bash cd doc pip install -r etc/requirements.txt .. warning:: Just follow the next step (copying conf.py from Doc project to your project) if that is your intention, otherwise skip it. Currently all projects should already have a conf.py file. Through the next step, this file and potential extensions in your project get overriden. Copy the conf.py file to your project folder where RST files have been kept: .. code-block:: bash cp docs/conf.py / Copy the static files to the project folder where RST files have been kept: .. code-block:: bash cp -r docs/_static/ / Build the documentation from within your project folder: .. code-block:: bash sphinx-build -b html / Your documentation shall be built as HTML inside the specified output folder directory. You can use your Web Browser to open and check resulting html pages in the output folder. .. note:: Be sure to remove the `conf.py`, the static/ files and the output folder from the `/docs/`. This is for testing only. Only commit the rst files and related content. .. _building-all-documentation: All Documentation ----------------- To build the all documentation under doc/, follow these steps: Install `tox `_. .. code-block:: bash sudo pip install tox Download the DOC repository. .. code-block:: bash git clone http://gerrit.onap.org/r/doc Build documentation using tox local environment & then open using any browser. .. code-block:: bash cd doc tox -elocal firefox docs/_build/html/index.html .. note:: Make sure to run `tox -elocal` and not just `tox`. This updates all submodule repositories that are integrated by the doc project. There are additional tox environment options for checking External URLs and Spelling. Use the tox environment options below and then look at the output with the Linux `more` or similar command scan for output that applies to the files you are validating. .. code-block:: bash tox -elinkcheck more < docs/_build/linkcheck/output.txt tox -espellcheck more < docs/_build/spellcheck/output.txt