organice documentation
Table of Contents
- 1. General
- 2. Installation
- 3. Usage
- 4. Customization
- 5. Development
- 6. Deployment
- 7. Contrib
- 8. Capture templates
- 9. Bookmarklets
- 10. Siri integration
- 11. Comparison
- 12. Attributions
- 13. FAQ
- 14. Development
- 15. Building this documentation
- 16. Contribute to organice
- 17. Code of Conduct
organice - /'ษ:gษnaษชz/
organice organizes Org files nicely!
1 General
Code repository: https://github.com/200ok-ch/organice
Community chat:
- #organice on IRC Libera.Chat
- #organice:matrix.org on Matrix
Creating and maintaining organice is made possible by all the volunteering contributors and sponsors. If you enjoy using organice, and want to sponsor the development of Free and Open Source software, you can do so with Github Sponsors and Patreon.๐
1.2 What does this project do?
organice is an implementation of Org mode without the dependency of Emacs. It is built for mobile and desktop browsers and syncs with Dropbox, GitLab, and WebDAV.
At 200ok, we run an instance of organice at https://organice.200ok.ch, which is open for anyone to use! organice does not have a back-end (it's just a front-end application, which uses different back-end storage providers). We don't store any kind of data on our servers - we also don't use analytics on organice.200ok.ch.
1.3 Why is this project useful
Emacs is great, but it's desktop software. For users who want to access or edit their Org mode files whilst on the go, organice is a great choice.
1.4 Introduction
If you prefer a video to some text, we've got you covered! For EmacsConf 2019, we've created a 10 minute introductory video into the rationale and usability of organice.
You can watch it on:
2 Installation
organice is a web application. You can use it from any browser. On iOS and Android, you can install organice to your homescreen. When started from there, it will run in full-screen as a Progressive Web Application (PWA) which will add offline capabilities (see chapters progressive web app and offline support).
To install organice to the homescreen follow these platform specific instructions:
iOS:
Open organice in Mobile Safari. Tap the "share" button and select "Add to Home Screen".
Android:
The exact procedure may differ depending on your browser and Android version. If you discover improvements to the following procedure, please let us know!
First open the organice web page in your mobile browser.
- On Chrome, tap the "menu" button (three vertically stacked dots) and select "Add to homescreen".
- On Firefox, "menu" button (three vertically stacked dots) and select "Install". If you don't have that option, you may have one to tap the home icon with the plus sign inside it which is immediately to the right of the URL in the address bar.
- Other browsers may have a similar procedure to one of these.
At this point, most browsers will present a popup banner with the option to "Add to homescreen" or "Install".
By default, when you start organice, it will display your root file directory. If you prefer to display a specific Org file instead, you can select it in the file settings.
3 Usage
3.1 Current restrictions/expectations of organice
"Current" means we're working hard on removing the following restrictions and expectations.
- organice understands only a few in-buffer settings (see Supported
in-buffer configuration)
- Other in-buffer settings are imported and re-exported but are not editable with organice.
- Other content before the first headline is imported and re-exported, but invisible and currently not editable with organice.
- After potential in-buffer settings, your Org file has to begin with a headline.
Apart from these restrictions, organice is very robust in reading and editing your Org file and not breaking any of it. We're having users with 10'000 lines in their files including all kinds of native Org functionality - and even these files work just fine in organice!
Generally, when working with distributed Org files, we're recommending to put them under version control and to check for bugs and racing conditions between clients.
Please file an issue if you find additional restrictions, expectations or bugs that you wouldnโt have expected.
3.1.1 Background information
organice has a custom parser for Org files. It works quite fine and has unit tests to prove it. One of the quality goals for the parser is that when it parses and re-exports an Org file, it should not change the original file. Not seeing unrelated diffs is important for the productivity of the user. It sounds trivial, but lots of alternative products do not live up to this expectation.
Writing a parser for a complex syntax like Org mode in custom code is hard. Therefore, we are in the process of implementing a proper EBNF based parser and a set of tests behind that. If you're interested, please check it out: https://github.com/200ok-ch/org-parser
The strategy we're using with regard to the parser is this:
- Keep improving the existing custom parser for new features and make bug fixes as long as the new one isn't ready.
- In parallel, work on the new one until there is feature parity between both parsers.
- When the new one is finished, integrate it into organice.
3.2 Progressive Web App
organice can run as a PWA (Progressive Web App) - see the installation instructions and does have offline support. From your home screen, organice will start up in full screen and it will use a Service Worker to cache the application. On a desktop browser, the Service Worker will be used automatically. This is implemented using the Create React App Progressive Web App functionality which enables the following features:
- All static assets are cached so that organice loads fast on subsequent visits, regardless of network connectivity.
- Updates are downloaded in the background.
- organice works regardless of network state, even if offline.
- On mobile devices, organice can be added directly to the user's home screen, app icon and all.
Following that, if you start modifying your Org file when offline, organice will recognize that you are offline and queue up the synchronization until you are online again.
organice also understands when it's local Org file is outdated compared to the upstream file and will ask you what you want to do - pull the one from the synchronization back-end, push the one from organice or cancel. This happens when you made changes to your file on at least two machines at the same time without synchronizing them in the meantime. For this, we recommend putting your Org file under version control which is the idiomatic solution for changing text based files on multiple machines in parallel.
3.3 Offline Support
Additionally to the offline support provided through implementing organice as a progressive web app (see above) organice has the following offline capabilities:
- Every file opened in organice will automatically be cached on your
device (through
localStorage
). - When visiting the file, again, it will immediately be loaded from the local storage and then loaded from the remote back-end.
- That makes loading and switching between files instant and gives you the ability to work on multiple files when being offline.
3.4 Multi file support
Agenda, Search, Task List, Refile and Capture Templates have the ability to work on multiple files. You can adjust the behavior for these on a file per file basis by creating "file settings" in the settings menu. Multi file support works well with the offline capabilities documented in progressive web app and offline support.
4 Customization
4.1 General
Since organice implements Org mode, one might wonder if we plan to duplicate the Emacs configuration strategy. In Emacs Org mode, there's more than 650 variables for customization - and on top of that, there's often two ways to configure things:
- Using elisp
- Using in-buffer settings
Modifying Org behavior using elisp (variables) is certainly mighty and powerful. However, the goal of organice is not to clone Emacs in full. In fact, it could be argued that this is not possible. Emacs being a LISP machine has inherent power that cannot be brought to a web application. Instead, the goal is to make Org mode accessible on smartphones and for non-Emacs users. For both use-cases, elisp variable configuration is not an idiomatic or ergonomic option.
organice implements this customization strategy:
- Use in-buffer settings where appropriate
- Build custom and mobile friendly user interfaces where appropriate
- For example capture templates
4.2 Supported in-buffer configuration
4.2.1 In-buffer settings
#+TODO
#+TYP_TODO
#+SEQ_TODO
4.2.2 #+STARTUP:
options
nologrepeat
: Do not record when reinstating repeating item
4.2.3 Drawer properties
logrepeat
andnologrepeat
: Whether to record when reinstating repeating item
:PROPERTIES: :LOGGING: logrepeat :END:
4.3 Themes / Color scheme / Dark Mode / Light Mode
organice bundles several popular color themes, each in light mode
and dark mode
.
If you've set up a color scheme preference in your operating system,
organice will honor this preference. It uses the
prefers-color-scheme
media query for this. Here, you can see if your
browser supports this media query: https://caniuse.com/?search=prefers-color-scheme
If you change your color scheme preference directly within organice, this naturally overrides your operating system preference. The color schemes in organice are implemented in a strategy pattern, so that adding new themes is quite easy.
These themes come bundled with organice:
4.3.1 Solarized
4.3.2 One
4.3.3 Gruvbox
4.3.4 Smyck
4.3.5 Code
4.4 Other customizations
For some customizations, organice exposes a mobile friendly user interface. Please find them in the 'settings' view (cogs icon in the header on the right).
5 Development
organice is built with React and Redux. It was bootstrapped with Create React App. The tests are written with React Testing Library. The internal data structures are written as immutable persistent data collections with the Immutable library.
5.1 Prerequisites
You will need a version of the Node.js engine installed which fulfills
the requirement stated in package.json
. If you don't already have
this installed, it is recommended to install it via nvm. The organice
repository already contains an .nvmrc
file, so once you have nvm
installed, the following commands should be sufficient:
nvm install nvm use
5.2 Setup
5.2.1 Installation of packages
To install the necessary packages, run:
yarn install --production=false
5.2.2 Setup any of the synchronization back-ends
organice can sync your Org files using Dropbox, GitLab, and WebDAV as back-ends.
If you want to develop a feature that needs synchronization, then you will have to set up any of those options. If you want to work on a feature that does not need synchronization, you can skip this step.
- WebDAV
organice has support for WebDAV and ships with a Docker container with a WebDAV server based on Apache. You can make use of that and use this WebDAV back-end for local development.
Having said that, if you're a Dropbox, then it's convenient to have a working setup for it if you want to test on files that are already in those back-ends. But it doesn't have to be a barrier, just to get started. And maybe you don't want to host your files with either of them anyway and use WebDAV all the way.
In any case, here's how to get running locally with a WebDAV setup.
- Dropbox or GitLab
To test against your own Dropbox or GitLab application, you'll need to create a
.env
file by copying .env.sample to just.env
.cp .env.sample .env
Then, fill in the blanks in
.env
with your Dropbox or GitLab credentials. More information about that is in the section Synchronization back-ends.
5.2.3 Running the application
yarn start
5.2.4 Running the tests:
yarn test
5.3 Testing
When you're developing a new feature and you want to manually test it, it's best to check it out in a Desktop browser and on your smartphone. This is how you do that:
5.3.1 Desktop
Run the application with yarn start
which will open organice in your
configured default browser. Alternatively, visit
http://localhost:3000
in the browser of your choice.
5.3.2 Smartphone
There are multiple options on how you can connect from your smartphone to your computer running organice.
When running organice with yarn start
, it will show you all the IPs
that the application server is bound to. One will be local to your
computer, one will be on your network (if you're connected to a LAN or
Wifi, that is).
If your smartphone has access to the same network, you can access it with the given IP address and port number.
If your new feature doesn't require a synchronization back-end, just
open the sample.org
file which doesn't require a login. You're good
to go.
Synchronizing with Dropbox or GitLab
If your new feature does require the Dropbox or GitLab synchronization back-end, there's an extra step you need to perform.
Both Dropbox and GitLab require a whitelist of domains that they
can be synchronized from. The whitelist for local domains is
exclusively short: http://localhost:3000
.
Hence, to be able to login from your phone to your dev instance of organice, you'll need to set up port forwarding. If you have a shell on your phone and an ssh client, you can do that with the following command:
ssh -L 3000:localhost:3000 user-dev-machine
If you don't have a shell on your phone, you can use a dedicated SSH application (like Termius).
5.4 Debugging Tests
Apart from the popular choice of console.log
-debugging, it's easy to
use Chrome or Chromium for debugging tests.
Place a debugger;
statement in any test, then run:
yarn test:dbg
This will start running your Jest tests, but pause before executing to allow a debugger to attach to the process.
Open the following in Chrome:
about:inspect
After opening that link, the Chrome Developer Tools will be displayed. Select inspect on your process and a breakpoint will be set at the first line of the react script (this is done to give you time to open the developer tools and to prevent Jest from executing before you have time to do so). Click the button that looks like a "play" button in the upper right hand side of the screen to continue execution. When Jest executes the test that contains the debugger statement, execution will pause and you can examine the current scope and call stack.
The "Create React App" upstream docs for this feature are here: https://create-react-app.dev/docs/debugging-tests/
5.5 Automatic deployments of reference instance
The productive reference instance of organice is deployed to
https://organice.200ok.ch/. On merging a pull request to master
,
code and documentation are automatically deployed to production.
For more complicated features (aka epics) that require more than one
pull request, there is a reference stage instance on
https://staging.organice.200ok.ch/. When working on epics, we follow
the popular nvie git branching model in that we successively create
feature branches against develop
until the epic is finished. On
merging a pull request to develop
, code and documentation are
automatically deployed to stage.
5.6 Contributions
Please see our contributor guidelines and our code of conduct.
5.7 Mockups
When discussing new UX, it is often helpful to add a mockup to the discussion to ensure that everyone is on the same page. When a new contributor suggests a UX change and it's not trivial, we will ask to include a mockup to the issue.
Of course, you're completely free to create such a mockup with whatever tool you feel comfortable with. A scan of a pen and paper will do, using Inkscape or Illustrator is nice and so on. If you don't have a personal preference, and want to get going quickly, you can use the mockup included in this repository. Find the file /doc/mockups/organice-mockup.excalidraw and upload it to the open source sketching tool excalidraw.com. There, make any changes you like, and export the result as either .png or .excalidraw and attach it to the original issue.
NB: The .excalidraw file can also be opened by any SVG capable tool like Inkscape.
6 Deployment
Since organice is a front-end only application, it can easily be deployed to any server capable of serving a static application.
Please note: If you want the hosted application to connect to Dropbox, GitLab or WebDAV, please read the section on Synchronization back-ends.
6.1 FTP
First create the production build locally: yarn run build
Note: Creating a build will actually make your REACT_APP_*
variables
from the .env
file available under process.env
even though it'll
be a front-end application.
And then upload to your web-server. Here's a sample script for your convenience:
HOST='your_ftp_server_host' USER='ftp_user' PASSWD='ftp_password' lftp $HOST <<END_SCRIPT user $USER $PASSWD mirror -R build/ quit END_SCRIPT exit 0
The reference instance (https://organice.200ok.ch), for example, is deployed via FTP. The full build script is in bin/compile\and\upload.sh.
6.2 Docker
organice is also available as a Docker image.
The docker image recognizes a couple of environment variables. For
example REACT_APP_WEBDAV_URL
prefills the URL field in the WebDAV
signin form. See docker-compose.yaml for an example how to use it.
A full list of such environment variables can be found in .env.sample.
The prefix REACT_APP_
has to be replaced with ORGANICE_
. The
naming should be pretty self explanatory.
6.2.1 With docker-compose
If docker-compose is installed, the following command downloads and runs the latest image automatically.
docker-compose up -d
The webserver is listening on port 5000 and can be reached here: http://localhost:5000
If you want to build the image yourself, use the
docker-compose-dev.yaml
file:
docker-compose -f docker-compose-dev.yaml up
6.2.2 Without docker-compose
If docker-compose
is not installed the command looks like this:
docker run -p 5000:5000 --name organice twohundredok/organice:latest
Again the webserver is listening on port 5000 and can be reached here: http://localhost:5000
6.3 Heroku
Assuming, you have an account and have installed the command line tools, deployment is as easy as:
heroku create
heroku config:set ON_HEROKU=1
git push heroku master
6.4 Synchronization back-ends
6.4.1 Dropbox
To configure your own instance of organice for Dropbox, please go to
the Dropbox developer console, create a new app and configure the
resulting clientId
in a newly created .env
file (analogous to
.env.sample
) as the value of the key REACT_APP_DROPBOX_CLIENT_ID
.
Make sure to add your own host URL (or http://localhost:3000/
for local development) as Redirect URI
.
Your dropbox app needs permission to read and write files.
6.4.2 WebDAV
- General
- More information
In the WebDAV FAQ, you'll find lots more information regarding WebDAV:
- A screencast of how organice works when logging in to a WebDAV server
- Documentation how on to setup your own WebDAV Server with Apache2 on Debian
- Documentation how to configure Nextcloud behind haproxy to allow WebDAV
- Documentation on Nextcloud sharing
6.4.3 GitLab
To configure your own instance of organice for GitLab, please create an OAuth application by going to GitLab's application settings for your profile and filling out the form with the following details:
- Name: "organice test" (or whatever you prefer)
- Redirect URI:
http://localhost:3000/
for local development, or whatever domain you are hosting it with. - Confidential: uncheck this
- Expire access tokens: leave checked
- Scopes:
api
only
Once filled out, click "save application" and keep this page open.
Then, create a new .env
file (analogous to .env.sample
) and set
the following variables:
REACT_APP_GITLAB_CLIENT_ID
: The value that GitLab provides forApplication ID
REACT_APP_GITLAB_SECRET
: The value that GitLab provides forSecret
.
You may also refer to GitLab's documentation for more information regarding OAuth applications, if interested.
6.4.4 Encryption
If you do not trust your data with third parties like Dropbox, you are free to use Gitlab (which is open-source) or host your own WebDAV server and take any number of precautionary measures.
For example, you can encrypt your data on disk. organice itself is just a front-end application, requires no server and has no tracking system. Therefore, the data within any organice instance (self hosted or not) is already only accessible to you, your browser and the network between your browser and your chosen back-end. Therefore, if you have a strong SSL certificate configured on your WebDAV server and organice instance, then organice will communicate securely via HTTPS to your server where your data is as secure as you make it. Then, your data will be encrypted and inaccessible to any third party.
Of course, security is hard. So the above statement is not a guarantee, but a guideline. You're responsible to ensure that the technologies employed (HTTPS, SSL, WebDAV, Browser, etc) are up to date and secure.
6.5 Routing
Whilst organice is a true Single Page Application (SPA) and therefore
has no back-end whatsoever, this does have an implication for
deployment with regard to routing. For routes like example.com/foo
to work, we need a little something extra. Within the context of a
running SPA, /foo
would be matched by the React Router and the
proper page would be rendered by JavaScript. When initially requesting
a route like that from the web server itself, the SPA is not running
yet and the web server itself wouldn't find a file called /foo
. It
would return a 404. The whole topic is explained in depth in this SO
answer: https://stackoverflow.com/a/36623117
For https://organice.200ok.ch we've opted to:
- Use the modern HTML5 history API with BrowserRouter
- Not configure a back-end for isomorphic routing, because it would complicate application and deployment unnecessarily (SEO is a non-issue for organice)
- Use good old Apache Webserver for hosting the compiled static assets
Therefore configuring a catchall is as easy as setting up a
.htaccess
file in the root of the organice folder containing:
RewriteEngine On RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR] RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d RewriteRule ^ - [L] RewriteRule ^ /index.html [L]
N.B.: If you're using WebDAV as a sync back-end, and the RewriteRule
is
allowed to apply to a WebDAV directory, it will break PUT requests to
upload new files! Here's documentation on how to configure both
features together correctly.
7 Contrib
organice aims to follow the 'batteries included' philosophy. For example, the documentation is rather extensive and includes wider topics than just its own functionality - for example it includes documentation on various tested deployment strategies.
However, the community regularly comes up with a whole range of
options on how to use organice more effectively for specific
use-cases. Sometimes, these options are generic enough so that the
maintainers take the functionality into core. Sometimes, it's not that
well suited to be added into core, but still is potentially very well
suited to a wider range of users. For that, organice follows the
contrib
model which many bigger projects use (i.e. Org mode) for
such contributions.
Please see the contrib folder for details.
8 Capture templates
organice supports capture templates by implementing a flexible mechanism using URL parameters. These three of the following parameters are required and must be URL encoded:
captureTemplateName
: the name of the capture template to use. This capture template must already exist in Settings > Capture templates.captureFile
: thepath
for Dropbox of the file in which to execute the capture template.captureContent
: the content you'd like to capture. This content will be placed at the cursor position if specified in the capture template (with%?
), or at the end of the template if it's not specified.
You can also specify additional custom variables for use in your
templates. They should be in the format captureVariable_<your custom
variable>
, and should also be URL encoded. In your capture template
they'd show up as %<your custom variable>
.
organice allows you to specify where the captured content will be inserted, via a "header path" which is a list of headers to match. If the list is empty, the content will be inserted at the end of the file, or the beginning if the prepend option is selected.
8.1 Examples
8.1.1 Simple: Capture a string
Say, you want to capture thoughts/todos as they occur to you. You might want to have a capture template to just get these things out of your head.
This makes for a good "Inbox" capture template:
Capture Template
* TODO %? %U
Example URL
Result
* TODO Read up on capture templates [2019-09-08 Sun 20:54]
8.1.2 With custom variable
If you want to add web pages to a reading queue (with a title, a capture date and a URL), this would be a good starting point:
Capture Template
* %? %u - URL: %mediaURL
Example URL
Result
* Play Emacs like an instrument [2019-09-08 Sun] - URL: https://200ok.ch/posts/2018-04-27_Play_Emacs_like_an_Instrument.html
9 Bookmarklets
Since organice is a web application, you can use the capture templates feature to create bookmarklets, of course! For example, if you want a bookmarklet to add the current page (title, capture date and URL) to your reading queue using this capture template, all you need is a little bit of JavaScript:
javascript:(function() { const {title} = document; const url = `https://organice.200ok.ch?captureTemplateName=Media&captureContent=${title}&captureFile=/org/media.org&captureVariable_mediaURL=${ window.location.href }`; window.open(url, "_blank"); })()
9.1 Bookmarklets Demo
9.1.1 iOS
This is what using a bookmarklet to capture a website looks like in iOS:
10 Siri integration
The organice capture mechanism integrates very nicely with the Siri Shortcuts feature in iOS, allowing you to use Siri to execute capture templates.
You can use this sample Shortcut to get started with this right away in iOS 12 or newer. Open the link on your iOS device and click "Get Shortcut". Then open up the Shortcuts app and edit the template by following the directions in the comments. Then record a Siri trigger and you're good to go!
11 Comparison
11.1 Beorg
Before starting work on organice, @munen (the original maintainer) used Beorg and donated to it multiple times, because he was very happy to have a good option to access Org files on my phone with it.
The important differences to him were:
- organice is FOSS which is very much in the spirit of Org whilst Beorg is proprietary
- organice is web based, so there is no lock-in to a specific device or OS
11.2 org-web
organice has a shared history with org-web. In fact, it is a friendly fork. organice differs from org-web in that:
- organice is a community driven project. See our
- Code of conduct
- Contributing guidelines
- Community chat: #organice on IRC Libera.Chat, or #organice:matrix.org on Matrix on Matrix
- organice has the commitment of a Swiss company (200ok llc: https://200ok.ch/)
behind it to continually work on it.
- 200ok has a strong track record in fostering Free and Open Source Software (https://200ok.ch/floss.html) and has co-organized EmacsConf 2019.
- That's also why organice is Free Software (with the strong AGPL-3.0 license) whereas org-web is Open Source (with The Unlicense).
- The continuous effort yields a certain power over time. At the time of writing this, organice has many times more commits (~2400 vs ~600) and contributors (36 vs. 9). Of course, quantity doesn't trump quality. However, many of the new contributors brought significant features and improvements, not just tiny patches.
- organice initially focused on becoming bug free - for example on parsing and exporting org files correctly.
- organice continues to evolve independently with its own feature set. For example, it has WebDAV support. For a list of all user visible changes, please see the changelog.
- organice is a project with equal focus on mobile as desktop browsers.
- org-web tracks users with Google Analytics. organice does not.
- organice has great documentation: https://organice.200ok.ch/documentation.html
11.2.1 What's new?
To see how organice differs from org-web, please consult the changelog which contains the user visible changes since forking.
11.2.2 Acknowledgment
We are extraordinarily grateful to DanielDe, the original creator!
We forked the project, because we have different visions on how to go forward. He envisions a mobile only solution, we think it's great to have organice be available to any browser to enable anyone on the go or any non-Emacs user easy access to Org files. Also, DanielDe thinks of org-web as his pet project whereas organice has the full power of 200ok llc behind it whilst building a strong self-sufficient community around it.
Thank you for all, DanielDe!๐
13 FAQ
13.1 Synchronization back-ends
13.1.1 Dropbox
- Permissions ("Full Dropbox" vs. "App folder")
The reference organice host https://organice.200ok.ch/ has the permission type "Full Dropbox". This serves many users well - especially those who have Org files in various locations that they want to access.
If you're a user and have concerns to give an application full access to your Dropbox, here are your options:
- You might not need to change anything after reading this: organice is a front-end application, there is no back-end and no monitoring whatsoever. So, when you login to Dropbox, only your browser will have access to your Dropbox. So, from a security perspective, you're not giving too much access to a server - your data cannot be seen by anyone else but you.
- organice is a free and open source application. Therefore, you can review the synchronization code anytime. For Dropbox, this is pretty straight forward since the synchronization code is less than 200loc.
- If you still have concerns, you can take full control! Since organice is free and open source, you are free to host it yourself. You can also create your own integration with Dropbox and select the permissions as you wish. Here's more documentation on deploying organice.
13.1.2 WebDAV
- Demo
Here's a demo of how organice works when logging in to a WebDAV server.
On the left, you see a branded version of OwnCloud, on the right you see organice. After logging in and making a minute change, you can see that the
last edited
timestamp in OwnCloud changes. Also, we're verifying the change directly in Emacs using Emacs and you can see the change has also been synchronized to my local machine. - WebDAV test server
For testing purposes, we use a Docker image with a proven and well documented server: Apache2 running on Debian. You can build the Docker image yourself - and customize the Apache configuration to your needs:
docker build -f doc/webdav/Dockerfile -t apache-webdav .
or with docker-compose:
docker-compose build apache-webdav
Then run the Apache2 WebDAV server with:
docker run -dit --name apache-webdav-app -p 8080:80 apache-webdav
or with docker-compose:
docker-compose up apache-webdav
On your host machine, you can now login with any WebDAV client using the URL
http://localhost:8080/webdav
. There is no authentication configuration, so any user account works (including omitted user accounts). It goes without saying that if you wanted to use this for production, please enable authentication. Within the test image, you'll find the sample.org file, so you can get started developing and testing right away.For testing WebDAV outside of organice, and you're an Emacs user, you can open this link (
C-c C-o
). Then, you will get a TRAMP session with dired open and you'll see the sample file. You can interact with it like in any other dired buffer. Obviously, this link will not work when looking at the documentation in the browser, you'll have to open the file WIKI.org in Emacs.If you prefer a command line client, you could use cadaver. Install and use it like this:
sudo apt -y install cadaver cadaver http://localhost:8080/webdav/
- Bug reports
If you have any trouble connecting to WebDAV using organice, it could be your setup (please consult CORS and Gotchas with WebDAV). In any case, if you want to open a bug report, please document your issue by referencing how it doesn't work using the official WebDAV test server.
- Bug reports
- CORS
Since organice is a front-end application, it will login with JavaScript from within the browser - in turn the Cross-Origin Resource Sharing (CORS) headers must be set appropriately. If they are not set, you will not be able to login to your service from a browser. Alternatively, if you're using a server like Apache or Nginx, you can simply get around CORS by hosting organice on the same domain as your service.
Please note, that when your back-end does not set the correct CORS headers, organice cannot show you a really semantic error message on that. The reason is that browsers hide this information from JavaScript. You will simply get a network error. However, you can easily debug it yourself by looking into the JavaScript console. No worries, you don't have to be a (JavaScript) developer to find out about that - here's a screencast showing you how to do it:
- Gotchas with WebDAV
preflight request doesn't pass access control check
error
If you are getting an error like
Access to XMLHttpRequest at 'https://my.site/dav/' from origin 'https://organice.200ok.ch' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.
then something is wrong with your webserver config. You can check whether a CORS preflight check is returning the right headers via:
curl -v -X OPTIONS https://my.server/webdav/ # For the official organice Apache2 WebDAV test server: # curl -v -X OPTIONS http://localhost:8080/webdav/
The output should include lines like this:
... < HTTP/1.1 200 OK ... < Access-Control-Allow-Origin: * < Access-Control-Allow-Methods: GET,POST,OPTIONS,DELETE,PUT,PROPFIND < Access-Control-Allow-Headers: Origin,Accept,X-Requested-With,Content-Type,Access-Control-Request-Method,Access-Control-Request-Headers,Authorization,X-CSRF-Token,Depth < Access-Control-Allow-Credentials: true < Allow: OPTIONS,GET,HEAD,POST,DELETE,TRACE,PROPFIND,PROPPATCH,COPY,MOVE,LOCK,UNLOCK
If your server doesn't give a
200 OK
response, or if theAccess-Control-Allow-*
headers are missing, you may find these articles helpful:- Using Apache
RewriteEngine
If your WebDAV directory happens to be not only on the same webserver, but also within a subdirectory of the directory containing a
.htaccess
file containing aRewriteRule
that also applies to the WebDAV directory (for example like this), then you will need to create another.htaccess
file in the top-level WebDAV directory containing this:RewriteEngine Off
Otherwise any attempts to use WebDAV to upload new files via
HTTP PUT
requests will fall foul of the/index.html
rewrite rule above, resulting in a403 Forbidden
response.Another way to avoid this more selectively is to precede that rule with:
RewriteCond %{REQUEST_METHOD} !PUT
- Symlinks don't work
Unfortunately, symlink support never made it into Apache's
mod_dav
. - Bind-mounts of individual files don't work
In an Apache
mod_dav
context, unfortunately you can't use bind mounts of a single file instead of symlinks, becausemod_dav
attempts to write any changes to a file atomically, by first writing to a temporary file and then atomically renaming it to the target file, and Linux prevents renaming to bind mounts with aDevice or resource busy
error. HTTP PUT
requests fail with403 Forbidden
As mentioned in the section routing, you should avoid having
mod_rewrite
rules apply to (PUT
) requests in the WebDAV directories.
- Configuring Nextcloud behind haproxy to allow WebDAV
If you're running Nextcloud behind haproxy it's entirely possible to use it with organice using WebDAV. …it's just a little bit convoluted.
The first part is the haproxy config. It should look a little bit like this:
frontend www acl host_nextcloud hdr(host) nextcloud.example.org acl path_nextcloud_public_webdav path_beg /public.php/webdav # Because we need to inspect the path in the backend section we set a variable # containing the path. http-request set-var(txn.path) path # Because the OPTIONS requests from organice doesn't include authentication we # need to fake it. We can do that by redirecting all requests that satisfy these conditions: # # + host is Nextcloud # + path is for public webdav # + HTTP method is OPTIONS use_backend always200ok if host_nextcloud path_nextcloud_public_webdav METH_OPTIONS # haproxy doesn't really have a way of returning an arbitrary response, unless # you want to drop down to Lua. There's no need for that, though, as this works # perfectly fine. This backend doesn't have any servers attached, so it'll # always result in a 503. We override the 503 by setting a custom errorfile, # which incidentally looks just like an HTTP 200 response and contains all the # headers we need to satisfy a CORS request. backend always200ok mode http errorfile 503 /etc/haproxy/errors/200-ok.http # The Nextcloud server backend is configured here. We inject CORS headers if URL # starts with `/public.php/webdav`. backend nextcloud mode http option httplog acl is_webdav var(txn.path) -m beg /public.php/webdav http-response add-header Access-Control-Allow-Origin "*" if is_webdav http-response add-header Access-Control-Allow-Methods "GET,POST,OPTIONS,DELETE,PUT,PROPFIND" if is_webdav http-response add-header Access-Control-Allow-Headers "Origin,Accept,X-Requested-With,Content-Type,Access-Control-Request-Method,Access-Control-Request-Headers,Authorization,X-CSRF-Token,Depth" if is_webdav http-response add-header Access-Control-Allow-Credentials "true" if is_webdav server backend01 127.0.0.1:8001
The
errorfile
needs to look something like the below. Note that the text below has carriage returns (13
,o15
or0x0d
); these are required as per the HTTP RFC!HTTP/1.1 200 OK Cache-Control: no-cache Connection: close Content-Type: text/html Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET,POST,OPTIONS,DELETE,PUT,PROPFIND Access-Control-Allow-Headers: Origin,Accept,X-Requested-With,Content-Type,Access-Control-Request-Method,Access-Control-Request-Headers,Authorization,X-CSRF-Token,Depth Access-Control-Allow-Credentials: true <html><body><h1>200 Stuff is good!</h1> Something something dark side. </body></html>
- Nextcloud sharing
In order to share a document using WebDAV you might be inclined to try to follow the official documentation, but it can be a tad confusing. Here's the executive summary for how to share things from Nextcloud using WebDAV:
- share a link to a folder/file
- remove everything but the token from the link; the token matches
/[a-zA-Z0-9]+$/
(hit the button right of "Share link" if using the web interface) - use these details when logging in:
- URL
- https://nextcloud.example.org/public.php/webdav
- Username
- the token, e.g.
ed65Fxw9Bz3MTn3
- Password
- if you've set a password for the shared folder, here's where you input it
- WebDAV Config
If you want to self-host organice and want to configure a default WebDAV server for your instance, you can by setting the variable
REACT_APP_WEBDAV_URL
in the.env
file.
13.2 Can we add feature X from plugin Y?
organice is an implementation of Org mode (see What does this project do?). Therefore, it is important that the changes in the markup made by organice are 100% compatible with Org mode itself.
Hence, if feature X from plugin Y can be implemented in a compatible way, and the feature follows the contribution guideline, then: Yes, the feature can be added to organice.
14 Development
14.1 Architecture Decision Records
14.1.1 adr-000-template
* TITLE <short present tense imperative phrase, less than 50 characters, like a git commit message.> ** Status <proposed, accepted, rejected, deprecated, superseded, etc.> ** Context # <what is the issue that we're seeing that is motivating this decision # or change.> ** Decision # <what is the change that we're actually proposing or doing.> ** Consequences # <what becomes easier or more difficult to do because of this change.>
14.1.2 adr-001-use-adrs
- Architecture Decision Record: Use ADRs
- Context
organice aims to be a helpful and accessible tool for many users and developers over years to come. Hence, practicing discipline of architecture is very important.
- We want to think deeply about all our architectural decisions, exploring all alternatives and making a careful, considered, well-researched choice.
- Even when the above statement does not hold true and we walk a pragmatic path, we want to be as transparent as possible in our decision-making process.
- We want to be able to revisit prior decisions to determine fairly if they still make sense, and if the motivating circumstances or conditions have changed.
- Decision
We will document every architecture-level decision for organice with an Architecture Decision Record. These are a well structured, relatively lightweight way to capture architectural proposals. They can serve as an artifact for discussion, and remain as an enduring record of the context and motivation of past decisions.
The workflow will be:
- A developer creates an ADR document outlining an approach for a particular question or problem. The ADR has an initial status of "proposed."
- The developers and maintainers discuss the ADR. During this period, the ADR should be updated to reflect additional context, concerns raised, and proposed changes.
- Once a maintainer has made a decision, the ADR can be transitioned to either an "accepted" or "rejected" state.
- Since the team working on organice is very agile and good architecture often emerges from actual code and spikes, it is very well possible for code to already be committed to the repository before the ADR is accepted or even addressed as an ADR.
- If a decision is revisited and a different conclusion is reached, a new ADR should be created documenting the context and rationale for the change. The new ADR should reference the old one, and once the new one is accepted, the old one should (in its "status" section) be updated to point to the new one. The old ADR should not be removed or otherwise modified except for the annotation pointing to the new ADR.
We will use the popular ADR template by Michal Nygard using this template.
- Status
Accepted
- Consequences
- Developers must write an ADR and submit it for review to make any architectural decision transparent – that is, any decision that affects the way organice is put together at a high level.
- We will have a concrete artifact around which to focus discussion, before finalizing decisions.
- If we follow the process, decisions will be made deliberately, by the maintainers.
- We will have a useful persistent record of why the system is the way it is.
- Context
14.1.3 adr-002-static-content-pages
- Architecture Decision Record: Static Content Pages
- Context
organice runs different kinds of pages:
- Static content like the Landing Page
- The actual application
These two do not share a whole lot in common. They have different needs in terms of CSS, Javascript, but also code quality. However, they run from the same
<App>
component which already has the assumption that it'll contain the actual application, not a content page like the Landing Page.This might seem convoluted and maybe there is a better solution. The seemingly only other solution is to
yarn eject
fromreact-scripts
, so that we could have multiple starting HTML files for multiple SPAs. This has the following downsides, though:react-scripts
is a great wrapper which we cannot use anymore.- Related to 1: Ejecting will create dozens of files which will have to be maintained manually.
- Decision
The downsides of ejecting seem larger than sharing the same
<App>
component and creating safeguards in the code where necessary to differentiate between the two use cases. - Status
Accepted
- Consequences
- Some safeguards in the code like:
- Setting a
landing-page
class dynamically in<Entry>
if the application is showing the Landing Page. - Making sure that static content pages do not
import
CSS code that pollutes the global namespace (reminder: CSS is not component local by default in React). Instead, we employ SCSS and scope all selectors under a unique identifier.
- Setting a
- Some safeguards in the code like:
- Context
14.1.4 adr-003-synchronization-back-ends
- Synchronization back-ends
- Context
Different users like to synchronize their files using different back-ends. Some prefer open standards and to host the required services themselves - others prefer to pay for a closed source service so that they don't have to worry about storage of their data themselves. The spread of strategies employed by these back-ends couldn't be further apart - some work in hunks and commits, some on files, etc.
To accommodate every personal synchronization strategy, organice should be easily extensible.
- Decision
organice employs the Strategy Pattern which is a design pattern made to do the 'same thing' (like storing and retrieving files) with 'different strategies' (like employing different APIs for Dropbox, GitLab, and WebDAV).
- Status
Accepted
- Consequences
Implementing a client for a new synchronization back-end is quite easy:
- Take any of the existing back-ends as a template. If you already happen to know how the API of your new synchronization back-end works, you can pick the closest one. Ultimately, it doesn't make a big difference from where you start off.
- Implement the 8 functions defined by the strategy pattern:
isSignedIn
getDirectoryListing
getMoreDirectoryListing
updateFile
createFile
getFileContentsAndMetadata
getFileContents
deleteFile
- That's it๐
- Context
15 Building this documentation
This comprehensive documentation is an aggregation of multiple files
which all reside in the organice code repository (README.org
,
WIKI.org
, CONTRIBUTING.org
, and CODE_OF_CONDUCT.md
).
To build the documentation locally, run make docs
.
Building the documentation and uploading it to https://organice.200ok.ch/documentation.html is part of the CI/CD workflow. The actual compilation happens here and the result gets uploaded here.
16 Contribute to organice
First off, thank you for considering contributing to organice. It's people like you that make organice such a great tool!
16.1 Did you find a bug or do you want to implement a new feature?
What do we consider a bug? A bug is when an existing feature is broken.
A feature request is to ask for change in an existing feature or to ask for a completely new feature.
Hence, if your request is about a feature that Emacs Org mode has, but organice lacks, this is a feature request, not a bug.
- Ensure the bug or feature was not already reported by searching on GitHub under Issues.
- Ensure that the the bug you want to report is not known behavior by reading the readme and the wiki.
- If you're unable to find an open issue addressing the problem, open
a new one.
- If you want to report a bug or request a feature, please use the "Bug report" or "Feature request" ticket and fill out the asked for metadata.
- If your request is neither about a bug nor feature, please do open a blank issue.
16.2 How we work with issues
One of our quality goals is to keep the amount of issues manageable. This means that a single person has to be able to groom the backlog (review the open issues) in a reasonable amount of time.
These are the steps we take to ensure a high quality in the backlog:
- We only keep issues open on a longer term basis that fulfill one of
the following criteria:
- Someone has committed to work on it or plausibly plans to do so in the future
- It is a bug or regression
- If someone files a bug or feature request which is either not clear to the maintainers or is likely not a bug, but known and documented behavior, we ask a question or link to the documentation. We also add a 'question' label to the issue. If the issue creator does not respond to the question within 5 days, we close the issue. If she does choose to respond at some point, she can re-open the issue.
If you just want to start a discussion about a new feature or a different kind of change, but do not plan on working on it yourself, you are very welcome to join the community chat: #organice on IRC Libera.Chat, or #organice:matrix.org on Matrix on Matrix. We are looking forward to seeing you there!
Alternatively, if you really want a change implemented, you can hire one of the maintainers to do the work. If you want to hire us to implement a change request, please send us an email to info@200ok.ch.
16.3 Contributing to the documentation
The documentation is generated from README.org
and various other files.
The section on building documentation explains how it is assembled.
16.4 Development collaboration
We have good quality assurance and an established workflow. This is it:
- Open a new GitHub pull request
- In the form of a User Story (As
<persona>
, [When I<state>
], I want<something>
, so that<measurable achievement>
- Ensure the PR description clearly describes the problem and solution. Include the relevant issue number if applicable.
- New Branch in Git
- Naming:
(feature|fix|chore)/issue-id/short-description
- If your PR includes a substantial change to the overall architecture, consider writing an architecture decision record.
- Develop / Test locally until tests pass
- We use prettier-eslint to format code, which combines both
prettier
andeslint --fix
for the best of both worlds. This repository includes.prettierrc.json
and.eslintrc.yml
with some configuration options that Prettier will use automatically. - Run
yarn prettier-eslint --write
to automatically format the whole codebase, oryarn prettier-eslint --list-different
to see which files don't currently match the repository's style. - If you're using Emacs, you can autoformat your source files: https://github.com/munen/emacs.d/#auto-formatting
- Please use
eslint
to ensure that your changes conform with our linting rules. You can runyarn eslint
to check for rule violations, and alsoyarn nibble
to tackle them one category at a time. yarn lint
will run both sets of linters.
- We use prettier-eslint to format code, which combines both
- Create Pull Request on Github to
master
- Check that the tests also pass on CI
- Core Team: Merge to
master
, deploy and accept the Issue on Github
16.4.1 Definition of Done
An issue is done when:
- Functionality has been implemented
- Functionality has been verified
- Surrounding main functionality has been regression tested
- Code has been reviewed
- Proper code style
- Code has tests (acceptance tests for user visible changes, otherwise at least unit tests)
- Follows clean code guidelines and architecture best practices
- Code has been documented
- Code has been merged
- Tested on supported browsers
Clean code guidelines
- Create small and highly cohesive modules
- Avoid long modules and classes
- Extract modules to separate responsibilities
- Create small methods or functions
- Avoid long methods or functions
- Extract methods or functions to separate responsibilities
- Do one thing
Thanks!๐๐
organice Team
200ok llc and all contributors
17 Code of Conduct
17.1 Our Pledge
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
17.2 Our Standards
Examples of behavior that contributes to creating a positive environment include:
- Using welcoming and inclusive language
- Being respectful of differing viewpoints and experiences
- Gracefully accepting constructive criticism
- Focusing on what is best for the community
- Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
- The use of sexualized language or imagery and unwelcome sexual attention or advances
- Trolling, insulting/derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or electronic address, without explicit permission
- Other conduct which could reasonably be considered inappropriate in a professional setting
17.3 Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
17.4 Scope
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
17.5 Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at info@200ok.ch. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
17.6 Attribution
This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html