Blog Video Cover (test not working this field on form)

Hi,

Thinking in loud voice. Nothing serious. This intents to be an uzta-based user-friendly plugin to carry data from old Loreas to Hydra.

 

Abstract:

 

Main goal here is to build a plugin named elgg-uzta with only three main features:

1 – Offer a form to set credentials and Lorea’s backup target url.

2 – Offer a checkable list of Uzta’s ontologies.

3 – Offer a downloadable compressed packages list for a given login, and offer to import it to Hydra. (take a look to elgg_file_takeout plugin)

 

Files:

 

/manifest.xml

/start.php

/actions/

– – – – /backup_session.php

– – – – /download_order.php

– – – – /downloadable_packages.php

/lib

– – – – /

/views/default/form/

– – – – /backup_session.php

– – – – /download_order.php

– – – – /downloadable_package.php

 


Specs:

 

1. Entities

2. Views

3. Actions

 


1. Entities:

 

1. a) backup_session ( username, password, url, flags )

Flags: target [“download”, “hydra”]

 

1. b) download_order ( backup_session, ontologies_list, order_timestamp )

1. b.1) ontologies_list:

  • 1. b.1.i) Group:
    # Type Codes for Search API:
    # 'file', 'page', 'assembly', 'videolist_item', 'bookmarks', 'task'
    # 'groupforumtopic'(discussion), 'album', 'blog', 'dokuwiki'

    group_ontology_to_query_strings = {
            'assemblies': 'assembly',
            'discussions': 'groupforumtopic',
            'tasks': 'task',
            'photos': 'album',
            'files': 'file',
            'videos': 'videolist_item',
            'bookmarks': 'bookmarks',
            'blog_entries': 'blog',
            'pages': 'page',
            'wiki': 'dokuwiki'
    }
  • 1. b.1.ii) User:
    # Type Codes for Search API:
    # 'file', 'page', 'thewire', 'videolist_item', 'bookmarks', 'task'
    # 'album', 'blog'

    user_ontology_to_query_strings = {
            'wires': 'thewire',
            'tasks': 'task',
            'photos': 'album',
            'files': 'file',
            'videos': 'videolist_item',
            'bookmarks': 'bookmarks',
            'blog_entries': 'blog',
            'pages': 'page'
    }

 

1. c) downloadable_package ( username, dataset_xml, creation_timestamp, download_link )

 


2. Views:

 

2. a) Backup_session filling form.

Three textboxes and a submit button. Lorea’s urls box could be a dropbox.

Tip: login page example.

 

2. b) Ordering downloads form.

Set in columns every checkable item on group, user, blogs, etc. ontologies. Maybe sync with current on going process to avoid multithreading. See also [PLUGIN] lorea-suicide to get a full user dataset.

 

2. c) Visualize available downloads and cron deleting them so we don’t run out storage quote.

Tip: item list like this.

 


3. Actions

 

3. a) Process backup_session form.

3. a.1) Just store entity and launch customize process screen.

 

3. b) Process download_order form.

 

3. b.1) Set a link with Uzta by running a Python script from PHP.

Take a look to Uzta’s readme:

./uzta.py -z yourusername yourpassword https://the.lorea.seed/

… or work with Elgg API through EXPORT method:

Variables

$entity = $vars[‘entity‘]
$export = new stdClass
$exportable_values = $entity->getExportableValues()
foreach($exportable_values as
$v) $export
url = $entity->getURL()
global $jsonexport
$jsonexport [$entity->getType()][$entity->getSubtype()][] = $export

3. b.2) Read target flag to set results destination.

While generating a zip file would be inmediat task thanks to Uzta, importing data to Hydra may need integrity checking. This frontend point of view could be hardless than gettings original Elgg entities through webservices, couldn’t it? May help ng-elgg browser API JSON map to send through REST API?

Fnordian warning from writer:

This point about importing data on user-decission way could be a killing point. Meaning that rebuilding in Hydra user blog entries would not offer too much difficulty but thinking in other content like discussions or assemblies may need missing users (should they be ghosted? should they be pre-created waiting for them to come?) or any needed entities. Always better to background massive 1Tera databases by admins. So… But also a kind of individul travelling through Lorea seeds, carring with oneself desired data would worth the effort…

 

3. b.3) Import to Hydra a Uzta generated package.

Proceed as Uzta by launching “writing” threads as “reading” orders.

 

3. b. 3.i) Readings

# Group read functions

def read_group_decisions(url, s):
    """
    Take url and return decision string from
    decision page. url => decision string
    """
    print('Original decision URL: ' + url)
    soup = bs(s.get(url).text, 'html5lib')
    decision_title = soup.select('h2')[0].text
    decision_author = soup.select('.elgg-image-block .elgg-body .elgg-subtext a')[0].text
    decision_content = soup.select('.elgg-main .elgg-output')[0].text
    try:
        decision_proposal_author = soup.select('.crud-children .elgg-image-block .elgg-body .elgg-subtext a')[0].text
        decision_proposal_content = soup.select('.crud-children .elgg-output')[0].text
    except:
        decision_proposal_author = ''
        decision_proposal_content = ''
    decision_comments = soup.select('.decision-comments .elgg-list li .mbn')
    decision_comment_authors = [x.select('a')[0].text for x in decision_comments]
    decision_comment_contents = [x.select('.elgg-output')[0].text for x in decision_comments]
    ordered_comments = OrderedDict([OrderedDict([('author', a), ('comment', c)]) for a, c in zip(decision_comment_authors, decision_comment_contents)])
    return OrderedDict([('title', decision_title),
                        ('author', decision_author),
                        ('content', decision_content),
                        ('proposal_author', decision_proposal_author),
                        ('proposal_content', decision_content),
                        ('comments', ordered_comments)])


def read_group_assemblies(url, s)
def read_group_blog_entries(url, s)
def read_group_bookmarks(url, s)
def read_group_videos(url, s)
def read_group_discussions(url, s)
def read_group_tasks(url, s)
def read_image_name_and_content(url, s)
def read_group_photos(url, s)
def read_file_name_and_content(url, s)
def read_group_files(url, s)
def read_group_wiki_page(url, s)
def read_group_wiki(url, s)
def read_group_pages(url, s)

# User read functions

def read_user_wires(url, s)
def read_user_blog_entries(url, s)
def read_user_bookmarks(url, s)
def read_user_videos(url, s)
def read_user_files(url, s)
def read_user_photos(url, s)
def read_user_tasks(url, s)
def read_user_pages(url, s)

 

3. b.3.ii) Writtings

# Group write functions

def write_group_assemblies(assemblies, output_dir_name, dir_name):
    filebasename = os.path.join(output_dir_name,
                                dir_name,
                                'assemblies',
                                'assembly')
    for assembly, n in zip(assemblies, range(len(assemblies))):
        filename = filebasename+str(n)+'.txt'
        with open(filename, 'w') as fileio:
            ordered_dump(assembly, fileio)          


def write_group_discussions(entries, output_dir_name, dir_name)
def write_group_tasks(tasks, output_dir_name, dir_name)
def write_group_photos(album_tuples, output_dir_name, dir_name)
def write_group_files(file_tuples, output_dir_name, dir_name)
def write_group_videos(videos, output_dir_name, dir_name)
def write_group_bookmarks(bookmarks, output_dir_name, dir_name)
def write_group_blog_entries(blog_entries, output_dir_name, dir_name)
def write_group_wiki(wiki_pages_list, output_dir_name, dir_name)
def write_group_decisions(decisions, output_dir_name, dir_name)
def write_group_pages(pages, output_dir_name, dir_name)

# User writing functions

def write_user_wires(wires, output_dir_name, dir_name)
def write_user_blog_entries(entries, output_dir_name, dir_name)
def write_user_bookmarks(bookmarks, output_dir_name, dir_name)
def write_user_videos(videos, output_dir_name, dir_name)
def write_user_files(files, output_dir_name, dir_name)
def write_user_photos(photos, output_dir_name, dir_name)
def write_user_tasks(tasks, output_dir_name, dir_name)
def write_user_pages(pages, output_dir_name, dir_name)
3. c) Cron task to delete-clean available downloads.

 

Using field: downloadable_package.creation_timestamp to purgue outdated packages.

  • psy

    psy hace 8 días

    inside this file: 29-11-2015_lorea_aleph.tar.gz.pgp

    will be as folders, all the “entities” and their “objects”, ordered by folders.. an example:

    +alpeh
    – Activity
    – Blogs
    – Bookmarks
    – Files
    – […]

    this can be done easily with a code connected to Elgg API, calling to “export” specific tables on database.

    with this, unhabitantes will have a folder with their data, organized on same way as network. this can open future doors to P2P sync file magics…

  • aleph

    aleph hace 37 minutos

    Why we have forked “file_takeout” plugin:

    .

    Because it offers basic flow management.

    1. User profile menu.
    2. Process backup order.
    3. Zip file.
    4. Dowloand pagehandler.

     

    1. User profile menu,

    Where to set setting and order command.

    See:

    file_takeout/views/default/plugins/file_takeout/settings.php

    file_takeout/views/default/plugins/file_takeout/usersettings.php


    2. Process backup order…

    …. by building listing entities loops over html.

    See:

     file_takeout/file_takeout.php#L79

    function create_files_from_entities($entities, $entity_type, $subtype, $guid, &$zip, &$file_takeout_tmp_files) {


    3. Zip file.

    See:

    file_takeout/file_takeout.php#L202

    // Create the ZIP archive and make it available for download
    if ($guid_from_path != ‘file_takeout’) {
         $files = get_all_entities($guid_from_path, ‘file’);
         $blogs = get_all_entities($guid_from_path, ‘blog’);
         $pages = get_all_entities($guid_from_path, ‘page_top’);
        $bookmarks = get_all_entities($guid_from_path, ‘bookmarks’);
         if (count($files) > 0 || count($blogs) > 0 || count($pages) > 0 || count($bookmarks) > 0) {
              $area .= ‘<h3>’ . get_entity($guid_from_path)->name . ‘</h3>’;
              $area .= ‘<br><p>Zipping the following files…</p>’;
              $area .= ‘<ul>’;
             $archive_path = elgg_get_data_path() . $guid_from_path . ‘.zip’;
             if (file_exists($archive_path)) {
                   unlink($archive_path);
              }
              $zip = new ZipArchive;
              $res = $zip->open($archive_path, ZipArchive::CREATE);

    4. Offer download url:

    Store on user namespace all generated packages urls. Work around garbage collector as commented in main paper.

    See pagehandler on:

    /file_takeout/start.php#L25

     

    How we are going to rebase forked “file_takeout” plugin:

    1. Add cripto feature to zipped package.
    2. Set also in settings Lorea source url and OAuth2 consumer.
    3. Hack “generation process” action…
    4. Run 1.9 plugin on 1.12
    5. Remove other generation file type than html
    6. Finish crypto feature…
    7. Add tar compression capabilities.
    8. Refactor manifest.

     

     

    1.- Add cripto feature to zipped package.

    Use download settings to set criptography. See plugin views.

     

    2.- Set also in settings Lorea source url and OAuth2 consumer.

    …, can Lorea act as GNUStatus or Twitter and offer to register App?

    http://docs.elgg.org/wiki/OAuth

    http://learn.elgg.org/en/1.x/guides/web-services.html

    – Take a look to point 5 on road-map-lorea-hydra-preversion:

    5.
    … (16:25:58) p: si, tenemos webservices pero ahora lo tengo apagado. Todo lo de conectar y federar…
    (16:27:36) p:
    Webservices Pack 1.0  Adds different webservices functions to the Elgg (REST) API
    (16:27:37) p: tenemos una nueva mas nueva que la q venia con elgg core
    16:32:03) p: te acabo de activar la API de REST, he puesto un tema “light” y he activado que puedas registrarte, asi que deberias de poder seguir el proceso, sin pedirme nada

    ¿https://github.com/ColdTrick/ws_pack?

     

    3.- Hack “generation process” action…

    … to retrieve JSON data through Elgg API using Elgg.export.

    See: file_takeout/file_takeout.php#L57

    // Save a few lines of code and use this helper function to grab all Elgg entities by owner and subtype (file, blog, page_top, bookmarks)
    function get_all_entities($guid, $subtype) {
         $options = array(
              ‘type’ => ‘object’,
              ‘subtype’ => $subtype,
              ‘container_guid’ => $guid,
              ‘limit’ => ”,
         );
         return elgg_get_entities($options);
    }

    Use some kind of:

    Simple CURL-based URL fetch library with some XML parsin

    https://elgg.org/plugins/download/521643

    4.- Run 1.9 plugin on 1.12

    and see what happens.

     

    5.- Remove other generation file type than html

    … aspx and docx are not friend filetypes, aren’t they?

    6.- Finish crypto feature…

    … by calling elgg-pgp encrypt method.

    7.- Add tar compression capabilities.

    8.- Refactor manifest.

Anuncios