Merge pull request 'main' (#1) from rimu/pyfedi:main into main

Reviewed-on: https://codeberg.org/mtyton/pyfedi/pulls/1

FEDERATION.md

@@ -0,0 +1,39 @@
+# Federation
+
+## Supported federation protocols and standards
+
+- [ActivityPub](https://www.w3.org/TR/activitypub/) (Server-to-Server)
+- [WebFinger](https://webfinger.net/)
+- [Http Signatures](https://datatracker.ietf.org/doc/html/draft-cavage-http-signatures)
+- [NodeInfo](https://nodeinfo.diaspora.software/)
+
+## Supported FEPS
+
+- [FEP-67ff: FEDERATION.md](https://codeberg.org/fediverse/fep/src/branch/main/fep/67ff/fep-67ff.md)
+- [FEP-f1d5: NodeInfo in Fediverse Software](https://codeberg.org/fediverse/fep/src/branch/main/fep/f1d5/fep-f1d5.md)
+- [FEP-1b12: Group federation](https://codeberg.org/fediverse/fep/src/branch/main/fep/1b12/fep-1b12.md)
+- [FEP-5feb: Search indexing consent for actors](https://codeberg.org/fediverse/fep/src/branch/main/fep/5feb/fep-5feb.md)
+- [FEP-2677: Identifying the Application Actor](https://codeberg.org/fediverse/fep/src/branch/main/fep/2677/fep-2677.md)
+
+## Partially Supported FEPS
+
+- [FEP-c0e0: Emoji reactions](https://codeberg.org/fediverse/fep/src/branch/main/fep/c0e0/fep-c0e0.md)
+  - Treated as a `like` or `upvote`
+- [FEP-268d: Search consent signals for objects](https://codeberg.org/fediverse/fep/src/branch/main/fep/268d/fep-268d.md)
+  - `"searchableBy": "https://www.w3.org/ns/activitystreams#Public"` == `"indexable": true`, any other content sets `"indexable": false`
+  - `"searchableBy": ["https://alice.example/actor", "https://example.com/users/1/followers"]` is not currently supported.
+
+## ActivityPub
+
+- Communities are ['Group' actors](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-group)
+- Users are ['Person' actors](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-person)
+- Posts on Communities are one of the following:
+  - [Page](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-page)
+  - [Article](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-article)
+  - [Link](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-link)
+  - [Note](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-note)
+  - [Question](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-question)
+
+## Additional documentation
+
+See the `docs/activitypub_examples` for examples of [communities](./docs/activitypub_examples/communities.md), [users](./docs/activitypub_examples/users.md), and [posts](./docs/activitypub_examples/posts.md).

INSTALL.md

@@ -114,7 +114,8 @@ pip install -r requirements.txt
 
 ### Extra info
 
-* `SERVER_NAME` should be the domain of the site/instance. Use `127.0.0.1:5000` during development unless using ngrok.
+* `SERVER_NAME` should be the domain of the site/instance. Use `127.0.0.1:5000` during development unless using ngrok. Just use the bare
+domain name, without https:// on the front or a slash on the end.
 * `RECAPTCHA_PUBLIC_KEY` and `RECAPTCHA_PRIVATE_KEY` can be generated at https://www.google.com/recaptcha/admin/create (this is optional - omit to allow registration without RECAPCHA).
 * `CACHE_TYPE` can be `FileSystemCache` or `RedisCache`. `FileSystemCache` is fine during development (set `CACHE_DIR` to `/tmp/piefed` or `/dev/shm/piefed`)
 while `RedisCache` **should** be used in production. If using `RedisCache`, set `CACHE_REDIS_URL` to `redis://localhost:6379/1`. Visit https://yourdomain/testredis to check if your redis url is working.
@@ -171,6 +172,8 @@ flask run
 (open web browser at http://127.0.0.1:5000)          
 (log in with username and password from admin account)
 
+For development purposes, that should be enough - see ./dev_notes.txt for a few more bits and pieces. Most of what follows is for running PieFed in production.
+
 <div id="database-management"></div>
 
 ## Database Management
@@ -245,11 +248,13 @@ also [see this](https://pganalyze.com/blog/5mins-postgres-tuning-huge-pages).
 To assess whether to accept a registration application it can be helpful to know the country of the applicant. This can be
 automatically discovered by using [the ipinfo service](https://ipinfo.io/) - register with them to get an API token and put it into your .env file.
 
+If the search function is not returning any results, you need to [add some database triggers](https://codeberg.org/rimu/pyfedi/issues/358#issuecomment-2475019).
+
 <div id="background-services"></div>
 
 ### Background services
 
-Gunicorn and Celery need to run as background services:
+In production, Gunicorn and Celery need to run as background services:
 
 #### Gunicorn
 

app/__init__.py

@@ -32,7 +32,7 @@ def get_locale():
         return 'en'
 
 
-db = SQLAlchemy()  # engine_options={'pool_size': 5, 'max_overflow': 10} # session_options={"autoflush": False}
+db = SQLAlchemy(session_options={"autoflush": False})  # engine_options={'pool_size': 5, 'max_overflow': 10} # session_options={"autoflush": False}
 migrate = Migrate()
 login = LoginManager()
 login.login_view = 'auth.login'

app/admin/forms.py

@@ -56,6 +56,13 @@ class PreLoadCommunitiesForm(FlaskForm):
     communities_num = IntegerField(_l('Number of Communities to add'), default=25)
     pre_load_submit = SubmitField(_l('Add Communities'))
 
+class RemoteInstanceScanForm(FlaskForm):
+    remote_url = StringField(_l('Remote Server'), validators=[DataRequired()])
+    communities_requested = IntegerField(_l('Number of Communities to add'), default=25)
+    minimum_posts = IntegerField(_l('Communities must have at least this many posts'), default=100)
+    minimum_active_users = IntegerField(_l('Communities must have at least this many active users in the past week.'), default=100)
+    dry_run = BooleanField(_l('Dry Run'))
+    remote_scan_submit = SubmitField(_l('Scan'))
 
 class ImportExportBannedListsForm(FlaskForm):
     import_file = FileField(_l('Import Bans List Json File'))
@@ -75,7 +82,6 @@ class EditCommunityForm(FlaskForm):
     local_only = BooleanField(_l('Only accept posts from current instance'))
     restricted_to_mods = BooleanField(_l('Only moderators can post'))
     new_mods_wanted = BooleanField(_l('New moderators wanted'))
-    show_home = BooleanField(_l('Posts show on home page'))
     show_popular = BooleanField(_l('Posts can be popular'))
     show_all = BooleanField(_l('Posts show in All list'))
     low_quality = BooleanField(_l("Low quality / toxic - upvotes in here don't add to reputation"))
@@ -119,6 +125,7 @@ class EditTopicForm(FlaskForm):
     name = StringField(_l('Name'), validators=[DataRequired()], render_kw={'title': _l('Human readable name for the topic.')})
     machine_name = StringField(_l('Slug'), validators=[DataRequired()], render_kw={'title': _l('A short and unique identifier that becomes part of the URL.')})
     parent_id = SelectField(_l('Parent topic'), coerce=int, validators=[Optional()], render_kw={'class': 'form-select'})
+    show_posts_in_children = BooleanField(_l('Show posts from child topics'), validators=[Optional()])
     submit = SubmitField(_l('Save'))
 
 
@@ -207,6 +214,8 @@ class EditUserForm(FlaskForm):
     bot = BooleanField(_l('This profile is a bot'))
     verified = BooleanField(_l('Email address is verified'))
     banned = BooleanField(_l('Banned'))
+    ban_posts = BooleanField(_l('Ban posts'))
+    ban_comments = BooleanField(_l('Ban comments'))
     hide_type_choices = [(0, _l('Show')),
                          (1, _l('Hide completely')),
                          (2, _l('Blur')),

app/admin/routes.py

@@ -1,4 +1,5 @@
 import os
+import re
 from datetime import timedelta
 from time import sleep
 from io import BytesIO
@@ -10,14 +11,15 @@ from flask_babel import _
 from slugify import slugify
 from sqlalchemy import text, desc, or_
 from PIL import Image
+from urllib.parse import urlparse
 
 from app import db, celery, cache
-from app.activitypub.routes import process_inbox_request, process_delete_request
+from app.activitypub.routes import process_inbox_request, process_delete_request, replay_inbox_request
 from app.activitypub.signature import post_request, default_context
 from app.activitypub.util import instance_allowed, instance_blocked, extract_domain_and_actor
 from app.admin.forms import FederationForm, SiteMiscForm, SiteProfileForm, EditCommunityForm, EditUserForm, \
     EditTopicForm, SendNewsletterForm, AddUserForm, PreLoadCommunitiesForm, ImportExportBannedListsForm, \
-    EditInstanceForm
+    EditInstanceForm, RemoteInstanceScanForm
 from app.admin.util import unsubscribe_from_everything_then_delete, unsubscribe_from_community, send_newsletter, \
     topics_for_form
 from app.community.util import save_icon_file, save_banner_file, search_for_community
@@ -196,6 +198,7 @@ def admin_federation():
     form = FederationForm()
     preload_form = PreLoadCommunitiesForm()
     ban_lists_form = ImportExportBannedListsForm()
+    remote_scan_form = RemoteInstanceScanForm()
 
     # this is the pre-load communities button
     if preload_form.pre_load_submit.data and preload_form.validate():
@@ -210,65 +213,60 @@ def admin_federation():
         community_json = resp.json()
         resp.close()
 
-        # sort out the nsfw communities
+        already_known = list(db.session.execute(text('SELECT ap_public_url FROM "community"')).scalars())
-        safe_for_work_communities = []
+        banned_urls = list(db.session.execute(text('SELECT domain FROM "banned_instances"')).scalars())
-        for community in community_json:
-            if community['nsfw']:
-                continue
-            else:
-                safe_for_work_communities.append(community)
-
-        # sort out any that have less than 100 posts
-        communities_with_lots_of_content = []
-        for community in safe_for_work_communities:
-            if community['counts']['posts'] < 100:
-                continue
-            else:
-                communities_with_lots_of_content.append(community)
-
-        # sort out any that do not have greater than 500 active users over the past week
-        communities_with_lots_of_activity = []
-        for community in communities_with_lots_of_content:
-            if community['counts']['users_active_week'] < 500:
-                continue
-            else:
-                communities_with_lots_of_activity.append(community)
-
-        # sort out any instances we have already banned
-        banned_instances = BannedInstances.query.all()
-        banned_urls = []
-        communities_not_banned = []
-        for bi in banned_instances:
-            banned_urls.append(bi.domain)
-        for community in communities_with_lots_of_activity:
-            if community['baseurl'] in banned_urls:
-                continue
-            else:
-                communities_not_banned.append(community)
-
-        # sort out the 'seven things you can't say on tv' names (cursewords, ie sh*t), plus some
-        # "low effort" communities
-        # I dont know why, but some of them slip through on the first pass, so I just 
-        # ran the list again and filter out more
-        #
-        # TO-DO: fix the need for the double filter
         seven_things_plus = [
             'shit', 'piss', 'fuck',
             'cunt', 'cocksucker', 'motherfucker', 'tits',
             'memes', 'piracy', '196', 'greentext', 'usauthoritarianism',
             'enoughmuskspam', 'political_weirdos', '4chan'
             ]
-        for community in communities_not_banned:
+
-            for word in seven_things_plus:
+        total_count = already_known_count = nsfw_count = low_content_count = low_active_users_count = banned_count = bad_words_count = 0
-                if word in community['name']:
+        candidate_communities = []
-                    communities_not_banned.remove(community)
+
-        for community in communities_not_banned:
+        for community in community_json:
-            for word in seven_things_plus:
+            total_count += 1
-                if word in community['name']:
+
-                    communities_not_banned.remove(community)
+            # sort out already known communities
+            if community['url'] in already_known:
+                already_known_count += 1
+                continue
+
+            # sort out the nsfw communities
+            elif community['nsfw']:
+                nsfw_count += 1
+                continue
+
+            # sort out any that have less than 100 posts
+            elif community['counts']['posts'] < 100:
+                low_content_count += 1
+                continue
+
+            # sort out any that do not have greater than 500 active users over the past week
+            elif community['counts']['users_active_week'] < 500:
+                low_active_users_count += 1
+                continue
+
+            # sort out any instances we have already banned
+            elif community['baseurl'] in banned_urls:
+                banned_count += 1
+                continue
+
+            # sort out the 'seven things you can't say on tv' names (cursewords), plus some
+            # "low effort" communities
+            if any(badword in community['name'].lower() for badword in seven_things_plus):
+                bad_words_count += 1
+                continue
+
+            else:
+                candidate_communities.append(community)
+
+        filtered_count = already_known_count + nsfw_count + low_content_count + low_active_users_count + banned_count + bad_words_count
+        flash(_('%d out of %d communities were excluded using current filters' % (filtered_count, total_count)))
 
         # sort the list based on the users_active_week key
-        parsed_communities_sorted = sorted(communities_not_banned, key=lambda c: c['counts']['users_active_week'], reverse=True)
+        parsed_communities_sorted = sorted(candidate_communities, key=lambda c: c['counts']['users_active_week'], reverse=True)
 
         # get the community urls to join
         community_urls_to_join = []
@@ -279,25 +277,37 @@ def admin_federation():
 
         # make the list of urls
         for i in range(communities_to_add):
-            community_urls_to_join.append(parsed_communities_sorted[i]['url'])
+            community_urls_to_join.append(parsed_communities_sorted[i]['url'].lower())
 
         # loop through the list and send off the follow requests
         # use User #1, the first instance admin
+
+        # NOTE: Subscribing using the admin's alt_profile causes problems:
+        # 1. 'Leave' will use the main user name so unsubscribe won't succeed.
+        # 2. De-selecting and re-selecting 'vote privately' generates a new alt_user_name every time,
+        #    so the username needed for a successful unsubscribe might get lost
+        # 3. If the admin doesn't have 'vote privately' selected, the federation JSON will populate
+        #    with a blank space for the name, so the subscription won't succeed.
+        # 4. Membership is based on user id, so using the alt_profile doesn't decrease the admin's joined communities
+        #
+        # Therefore, 'main_user_name=False' has been changed to 'admin_preload=True' below
+
         user = User.query.get(1)
         pre_load_messages = []
         for community in community_urls_to_join:
             # get the relevant url bits
             server, community = extract_domain_and_actor(community)
+
             # find the community
             new_community = search_for_community('!' + community + '@' + server)
-            # subscribe to the community using alt_profile
+            # subscribe to the community
             # capture the messages returned by do_subscibe
             # and show to user if instance is in debug mode
             if current_app.debug:
-                message = do_subscribe(new_community.ap_id, user.id, main_user_name=False)
+                message = do_subscribe(new_community.ap_id, user.id, admin_preload=True)
                 pre_load_messages.append(message)
             else:
-                message_we_wont_do_anything_with = do_subscribe.delay(new_community.ap_id, user.id, main_user_name=False)
+                message_we_wont_do_anything_with = do_subscribe.delay(new_community.ap_id, user.id, admin_preload=True)
 
         if current_app.debug:
             flash(_('Results: %(results)s', results=str(pre_load_messages)))
@@ -308,6 +318,250 @@ def admin_federation():
 
         return redirect(url_for('admin.admin_federation'))
 
+    # this is the remote server scan
+    elif remote_scan_form.remote_scan_submit.data and remote_scan_form.validate():
+        # filters to be used later
+        already_known = list(db.session.execute(text('SELECT ap_public_url FROM "community"')).scalars())
+        banned_urls = list(db.session.execute(text('SELECT domain FROM "banned_instances"')).scalars())
+        seven_things_plus = [
+            'shit', 'piss', 'fuck',
+            'cunt', 'cocksucker', 'motherfucker', 'tits',
+            'memes', 'piracy', '196', 'greentext', 'usauthoritarianism',
+            'enoughmuskspam', 'political_weirdos', '4chan'
+        ]
+        is_lemmy = False
+        is_mbin = False
+
+
+        # get the remote_url data
+        remote_url = remote_scan_form.remote_url.data
+
+        # test to make sure its a valid fqdn
+        regex_pattern = '^(https:\/\/)(?=.{1,255}$)((.{1,63}\.){1,127}(?![0-9]*$)[a-z0-9-]+\.?)$'
+        result = re.match(regex_pattern, remote_url)
+        if result is None:
+            flash(_(f'{remote_url} does not appear to be a valid url. Make sure input is in the form "https://server-name.tld" without trailing slashes or paths.'))
+            return redirect(url_for('admin.admin_federation'))
+
+        # check if it's a banned instance
+        # Parse the URL
+        parsed_url = urlparse(remote_url)
+        # Extract the server domain name
+        server_domain = parsed_url.netloc
+        if server_domain in banned_urls:
+            flash(_(f'{remote_url} is a banned instance.'))
+            return redirect(url_for('admin.admin_federation'))
+
+        # get dry run
+        dry_run = remote_scan_form.dry_run.data
+
+        # get the number of follows requested
+        communities_requested = remote_scan_form.communities_requested.data
+
+        # get the minimums
+        min_posts = remote_scan_form.minimum_posts.data
+        min_users = remote_scan_form.minimum_active_users.data
+
+        # get the nodeinfo
+        resp = get_request(f'{remote_url}/.well-known/nodeinfo')
+        nodeinfo_dict = json.loads(resp.text)
+
+        # check the ['links'] for instanceinfo url
+        schema2p0 = "http://nodeinfo.diaspora.software/ns/schema/2.0"
+        schema2p1 = "http://nodeinfo.diaspora.software/ns/schema/2.1"
+        for e in nodeinfo_dict['links']:
+            if e['rel'] == schema2p0 or e['rel'] == schema2p1:
+                remote_instanceinfo_url = e["href"]
+
+        # get the instanceinfo
+        resp = get_request(remote_instanceinfo_url)
+        instanceinfo_dict = json.loads(resp.text)
+
+        # determine the instance software
+        instance_software_name = instanceinfo_dict['software']['name']
+        # instance_software_version = instanceinfo_dict['software']['version']
+
+        # if the instance is not running lemmy or mbin break for now as 
+        # we dont yet support others for scanning
+        if instance_software_name == "lemmy":
+            is_lemmy = True
+        elif instance_software_name == "mbin":
+            is_mbin = True
+        else:
+            flash(_(f"{remote_url} does not appear to be a lemmy or mbin instance."))
+            return redirect(url_for('admin.admin_federation'))
+
+        if is_lemmy:
+            # lemmy has a hard-coded upper limit of 50 commnities
+            # in their api response
+            # loop through and send off requests to the remote endpoint for 50 communities at a time
+            comms_list = []
+            page = 1
+            get_more_communities = True
+            while get_more_communities:
+                params = {"sort":"Active","type_":"Local","limit":"50","page":f"{page}","show_nsfw":"false"}
+                resp = get_request(f"{remote_url}/api/v3/community/list", params=params)
+                page_dict = json.loads(resp.text)
+                # get the individual communities out of the communities[] list in the response and 
+                # add them to a holding list[] of our own
+                for c in page_dict["communities"]:
+                    comms_list.append(c)
+                # check the amount of items in the page_dict['communities'] list
+                # if it's lesss than 50 then we know its the last page of communities
+                # so we break the loop
+                if len(page_dict['communities']) < 50:
+                    get_more_communities = False
+                else:
+                    page += 1
+
+            # filter out the communities
+            already_known_count = nsfw_count = low_content_count = low_active_users_count = bad_words_count = 0
+            candidate_communities = []
+            for community in comms_list:
+                # sort out already known communities
+                if community['community']['actor_id'] in already_known:
+                    already_known_count += 1
+                    continue
+                # sort out any that have less than minimum posts
+                elif community['counts']['posts'] < min_posts:
+                    low_content_count += 1
+                    continue
+                # sort out any that do not have greater than the requested active users over the past week
+                elif community['counts']['users_active_week'] < min_users:
+                    low_active_users_count += 1
+                    continue
+                # sort out the 'seven things you can't say on tv' names (cursewords), plus some
+                # "low effort" communities
+                if any(badword in community['community']['name'].lower() for badword in seven_things_plus):
+                    bad_words_count += 1
+                    continue
+                else:
+                    candidate_communities.append(community)
+
+            # get the community urls to join
+            community_urls_to_join = []
+
+            # if the admin user wants more added than we have, then just add all of them
+            if communities_requested > len(candidate_communities):
+                communities_to_add = len(candidate_communities)
+            else:
+                communities_to_add = communities_requested
+
+            # make the list of urls
+            for i in range(communities_to_add):
+                community_urls_to_join.append(candidate_communities[i]['community']['actor_id'].lower())
+
+            # if its a dry run, just return the stats
+            if dry_run:
+                message = f"Dry-Run for {remote_url}: \
+                            Local Communities on the server: {len(comms_list)}, \
+                            Communities we already have: {already_known_count}, \
+                            Communities below minimum posts: {low_content_count}, \
+                            Communities below minimum users: {low_active_users_count}, \
+                            Candidate Communities based on filters: {len(candidate_communities)}, \
+                            Communities to join request: {communities_requested}, \
+                            Communities to join based on current filters: {len(community_urls_to_join)}."
+                flash(_(message))
+                return redirect(url_for('admin.admin_federation'))
+
+        if is_mbin:
+            # loop through and send the right number of requests to the remote endpoint for mbin
+            # mbin does not have the hard-coded limit, but lets stick with 50 to match lemmy 
+            mags_list = []
+            page = 1
+            get_more_magazines = True
+            while get_more_magazines:
+                params = {"p":f"{page}","perPage":"50","sort":"active","federation":"local","hide_adult":"hide"}
+                resp = get_request(f"{remote_url}/api/magazines", params=params)
+                page_dict = json.loads(resp.text)
+                # get the individual magazines out of the items[] list in the response and 
+                # add them to a holding list[] of our own
+                for m in page_dict['items']:
+                    mags_list.append(m)
+                # check the amount of items in the page_dict['items'] list
+                # if it's lesss than 50 then we know its the last page of magazines
+                # so we break the loop
+                if len(page_dict['items']) < 50:
+                    get_more_magazines = False
+                else:
+                    page += 1
+            
+            # filter out the magazines
+            already_known_count = low_content_count = low_subscribed_users_count = bad_words_count = 0
+            candidate_communities = []
+            for magazine in mags_list:
+                # sort out already known communities
+                if magazine['apProfileId'] in already_known:
+                    already_known_count += 1
+                    continue
+                # sort out any that have less than minimum posts
+                elif magazine['entryCount'] < min_posts:
+                    low_content_count += 1
+                    continue
+                # sort out any that do not have greater than the requested users over the past week
+                # mbin does not show active users here, so its based on subscriber count
+                elif magazine['subscriptionsCount'] < min_users:
+                    low_subscribed_users_count += 1
+                    continue
+                # sort out the 'seven things you can't say on tv' names (cursewords), plus some
+                # "low effort" communities
+                if any(badword in magazine['name'].lower() for badword in seven_things_plus):
+                    bad_words_count += 1
+                    continue
+                else:
+                    candidate_communities.append(magazine)
+
+            # get the community urls to join
+            community_urls_to_join = []
+
+            # if the admin user wants more added than we have, then just add all of them
+            if communities_requested > len(candidate_communities):
+                magazines_to_add = len(candidate_communities)
+            else:
+                magazines_to_add = communities_requested
+
+            # make the list of urls
+            for i in range(magazines_to_add):
+                community_urls_to_join.append(candidate_communities[i]['apProfileId'].lower())
+            
+            # if its a dry run, just return the stats
+            if dry_run:
+                message = f"Dry-Run for {remote_url}: \
+                            Local Magazines on the server: {len(mags_list)}, \
+                            Magazines we already have: {already_known_count}, \
+                            Magazines below minimum posts: {low_content_count}, \
+                            Magazines below minimum users: {low_subscribed_users_count}, \
+                            Candidate Magazines based on filters: {len(candidate_communities)}, \
+                            Magazines to join request: {communities_requested}, \
+                            Magazines to join based on current filters: {len(community_urls_to_join)}."
+                flash(_(message))
+                return redirect(url_for('admin.admin_federation'))
+
+        user = User.query.get(1)
+        remote_scan_messages = []
+        for community in community_urls_to_join:
+            # get the relevant url bits
+            server, community = extract_domain_and_actor(community)
+            # find the community
+            new_community = search_for_community('!' + community + '@' + server)
+            # subscribe to the community
+            # capture the messages returned by do_subscribe
+            # and show to user if instance is in debug mode
+            if current_app.debug:
+                message = do_subscribe(new_community.ap_id, user.id, admin_preload=True)
+                remote_scan_messages.append(message)
+            else:
+                message_we_wont_do_anything_with = do_subscribe.delay(new_community.ap_id, user.id, admin_preload=True)
+
+        if current_app.debug:
+            flash(_('Results: %(results)s', results=str(remote_scan_messages)))
+        else:
+            flash(
+                _('Based on current filters, the subscription process for %(communities_to_join)d of %(candidate_communities)d communities launched in background, check admin/activities for details',
+                  communities_to_join=len(community_urls_to_join), candidate_communities=len(candidate_communities)))
+
+        return redirect(url_for('admin.admin_federation'))
+
     # this is the import bans button
     elif ban_lists_form.import_submit.data and ban_lists_form.validate():
         import_file = request.files['import_file']
@@ -433,7 +687,7 @@ def admin_federation():
 
     return render_template('admin/federation.html', title=_('Federation settings'), 
                            form=form, preload_form=preload_form, ban_lists_form=ban_lists_form,
-                           current_app_debug=current_app.debug,
+                           remote_scan_form=remote_scan_form, current_app_debug=current_app.debug,
                            moderating_communities=moderating_communities(current_user.get_id()),
                            joined_communities=joined_communities(current_user.get_id()),
                            menu_topics=menu_topics(),
@@ -604,10 +858,8 @@ def activity_json(activity_id):
 def activity_replay(activity_id):
     activity = ActivityPubLog.query.get_or_404(activity_id)
     request_json = json.loads(activity.activity_json)
-    if 'type' in request_json and request_json['type'] == 'Delete' and request_json['id'].endswith('#delete'):
+    replay_inbox_request(request_json)
-        process_delete_request(request_json, activity.id, None)
+
-    else:
-        process_inbox_request(request_json, activity.id, None)
     return 'Ok'
 
 
@@ -678,7 +930,6 @@ def admin_community_edit(community_id):
         community.local_only = form.local_only.data
         community.restricted_to_mods = form.restricted_to_mods.data
         community.new_mods_wanted = form.new_mods_wanted.data
-        community.show_home = form.show_home.data
         community.show_popular = form.show_popular.data
         community.show_all = form.show_all.data
         community.low_quality = form.low_quality.data
@@ -735,7 +986,6 @@ def admin_community_edit(community_id):
         form.local_only.data = community.local_only
         form.new_mods_wanted.data = community.new_mods_wanted
         form.restricted_to_mods.data = community.restricted_to_mods
-        form.show_home.data = community.show_home
         form.show_popular.data = community.show_popular
         form.show_all.data = community.show_all
         form.low_quality.data = community.low_quality
@@ -818,7 +1068,8 @@ def admin_topic_add():
     form = EditTopicForm()
     form.parent_id.choices = topics_for_form(0)
     if form.validate_on_submit():
-        topic = Topic(name=form.name.data, machine_name=slugify(form.machine_name.data.strip()), num_communities=0)
+        topic = Topic(name=form.name.data, machine_name=slugify(form.machine_name.data.strip()), num_communities=0,
+                      show_posts_in_children=form.show_posts_in_children.data)
         if form.parent_id.data:
             topic.parent_id = form.parent_id.data
         else:
@@ -848,6 +1099,7 @@ def admin_topic_edit(topic_id):
         topic.name = form.name.data
         topic.num_communities = topic.communities.count()
         topic.machine_name = form.machine_name.data
+        topic.show_posts_in_children = form.show_posts_in_children.data
         if form.parent_id.data:
             topic.parent_id = form.parent_id.data
         else:
@@ -860,6 +1112,7 @@ def admin_topic_edit(topic_id):
         form.name.data = topic.name
         form.machine_name.data = topic.machine_name
         form.parent_id.data = topic.parent_id
+        form.show_posts_in_children.data = topic.show_posts_in_children
     return render_template('admin/edit_topic.html', title=_('Edit topic'), form=form, topic=topic,
                            moderating_communities=moderating_communities(current_user.get_id()),
                            joined_communities=joined_communities(current_user.get_id()),
@@ -962,7 +1215,7 @@ def admin_content_trash():
 
     page = request.args.get('page', 1, type=int)
 
-    posts = Post.query.filter(Post.posted_at > utcnow() - timedelta(days=3), Post.deleted == False).order_by(Post.score)
+    posts = Post.query.filter(Post.posted_at > utcnow() - timedelta(days=3), Post.deleted == False, Post.down_votes > 0).order_by(Post.score)
     posts = posts.paginate(page=page, per_page=100, error_out=False)
 
     next_url = url_for('admin.admin_content_trash', page=posts.next_num) if posts.has_next else None
@@ -1024,12 +1277,12 @@ def admin_content_deleted():
 
     posts = Post.query.\
         filter(Post.deleted == True).\
-        order_by(Post.posted_at)
+        order_by(desc(Post.posted_at))
     posts = posts.paginate(page=page, per_page=100, error_out=False)
 
     post_replies = PostReply.query. \
         filter(PostReply.deleted == True). \
-        order_by(PostReply.posted_at)
+        order_by(desc(PostReply.posted_at))
     post_replies = post_replies.paginate(page=replies_page, per_page=100, error_out=False)
 
     next_url = url_for('admin.admin_content_deleted', page=posts.next_num) if posts.has_next else None
@@ -1089,6 +1342,8 @@ def admin_user_edit(user_id):
     if form.validate_on_submit():
         user.bot = form.bot.data
         user.banned = form.banned.data
+        user.ban_posts = form.ban_posts.data
+        user.ban_comments = form.ban_comments.data
         user.hide_nsfw = form.hide_nsfw.data
         user.hide_nsfl = form.hide_nsfl.data
         if form.verified.data and not user.verified:
@@ -1122,6 +1377,8 @@ def admin_user_edit(user_id):
         form.bot.data = user.bot
         form.verified.data = user.verified
         form.banned.data = user.banned
+        form.ban_posts.data = user.ban_posts
+        form.ban_comments.data = user.ban_comments
         form.hide_nsfw.data = user.hide_nsfw
         form.hide_nsfl.data = user.hide_nsfl
         if user.roles and user.roles.count() > 0:

app/admin/util.py

@@ -8,8 +8,10 @@ from flask_babel import _
 
 from app import db, cache, celery
 from app.activitypub.signature import post_request, default_context
+from app.activitypub.util import extract_domain_and_actor
+
 from app.models import User, Community, Instance, Site, ActivityPubLog, CommunityMember, Language
-from app.utils import gibberish, topic_tree
+from app.utils import gibberish, topic_tree, get_request
 
 
 def unsubscribe_from_everything_then_delete(user_id):
@@ -124,5 +126,3 @@ def topics_for_form_children(topics, current_topic: int, depth: int) -> List[Tup
             result.extend(topics_for_form_children(topic['children'], current_topic, depth + 1))
     return result
 
-
-

app/api/alpha/routes.py

@@ -404,5 +404,62 @@ def alpha_emoji():
     return jsonify({"error": "not_yet_implemented"}), 400
 
 
+# HTML routes
+from flask import abort, render_template
+from app.utils import current_theme
+import os
+
+@bp.route('/api/alpha/', methods=['GET'])
+def get_alpha():
+    if not current_app.debug:
+        abort(404)
+
+    template_name = "index.html"
+
+    theme = current_theme()
+    if theme != '' and os.path.exists(f'app/templates/themes/{theme}/{template_name}'):
+        return render_template(f'themes/{theme}/{template_name}')
+    else:
+        return render_template(template_name)
 
 
+@bp.route('/api/alpha/auth/login', methods=['GET'])
+def get_alpha_auth_login():
+    if not current_app.debug:
+        abort(404)
+
+    template_name = "auth/login.html"
+
+    theme = current_theme()
+    if theme != '' and os.path.exists(f'app/templates/themes/{theme}/{template_name}'):
+        return render_template(f'themes/{theme}/{template_name}')
+    else:
+        return render_template(template_name)
+
+
+@bp.route('/api/alpha/auth/logout', methods=['GET'])
+def get_alpha_auth_logout():
+    if not current_app.debug:
+        abort(404)
+
+    template_name = "auth/logout.html"
+
+    theme = current_theme()
+    if theme != '' and os.path.exists(f'app/templates/themes/{theme}/{template_name}'):
+        return render_template(f'themes/{theme}/{template_name}')
+    else:
+        return render_template(template_name)
+
+
+@bp.route('/api/alpha/communities', methods=['GET'])
+def get_alpha_communities():
+    if not current_app.debug:
+        abort(404)
+
+    template_name = "list_communities.html"
+
+    theme = current_theme()
+    if theme != '' and os.path.exists(f'app/templates/themes/{theme}/{template_name}'):
+        return render_template(f'themes/{theme}/{template_name}')
+    else:
+        return render_template(template_name)

app/api/alpha/utils/site.py

@@ -20,9 +20,16 @@ def get_site(auth):
         user = None
 
     logo = g.site.logo if g.site.logo else '/static/images/logo2.png'
+    logo_152 = g.site.logo_152 if g.site.logo_152 else '/static/images/apple-touch-icon.png'
+    logo_32 = g.site.logo_32 if g.site.logo_32 else '/static/images/favicon-32x32.png'
+    logo_16 = g.site.logo_16 if g.site.logo_16 else '/static/images/favicon-16x16.png'
     site = {
       "enable_downvotes": g.site.enable_downvotes,
       "icon": f"https://{current_app.config['SERVER_NAME']}{logo}",
+      "icon_152": f"https://{current_app.config['SERVER_NAME']}{logo_152}",
+      "icon_32": f"https://{current_app.config['SERVER_NAME']}{logo_32}",
+      "icon_16": f"https://{current_app.config['SERVER_NAME']}{logo_16}",
+      "registration_mode": g.site.registration_mode,
       "name": g.site.name,
       "actor_id": f"https://{current_app.config['SERVER_NAME']}/",
       "user_count": users_total(),

app/cli.py

@@ -209,6 +209,11 @@ def register(app):
             db.session.execute(text('DELETE FROM "post_reply_vote" WHERE created_at < :cutoff'), {'cutoff': utcnow() - timedelta(days=28 * 6)})
             db.session.commit()
 
+            # Un-ban after ban expires
+            db.session.execute(text('UPDATE "user" SET banned = false WHERE banned is true AND banned_until < :cutoff AND banned_until is not null'),
+                               {'cutoff': utcnow()})
+            db.session.commit()
+
             # Check for dormant or dead instances
             try:
                 # Check for dormant or dead instances
@@ -219,8 +224,8 @@ def register(app):
                     if instance_banned(instance.domain) or instance.domain == 'flipboard.com':
                         continue
                     nodeinfo_href = instance.nodeinfo_href
-                    if instance.software == 'lemmy' and instance.version >= '0.19.4' and instance.nodeinfo_href and instance.nodeinfo_href.endswith(
+                    if instance.software == 'lemmy' and instance.version is not None and instance.version >= '0.19.4' and \
-                            'nodeinfo/2.0.json'):
+                            instance.nodeinfo_href and instance.nodeinfo_href.endswith('nodeinfo/2.0.json'):
                         nodeinfo_href = None
 
                     if not nodeinfo_href:
@@ -246,8 +251,10 @@ def register(app):
                         except Exception as e:
                             db.session.rollback()
                             current_app.logger.error(f"Error processing instance {instance.domain}: {e}")
+                            instance.failures += 1
                         finally:
                             nodeinfo.close()
+                        db.session.commit()
 
                     if instance.nodeinfo_href:
                         try:
@@ -266,6 +273,7 @@ def register(app):
                             current_app.logger.error(f"Error processing nodeinfo for {instance.domain}: {e}")
                         finally:
                             node.close()
+                        db.session.commit()
 
                     # Handle admin roles
                     if instance.online() and (instance.software == 'lemmy' or instance.software == 'piefed'):
@@ -275,12 +283,14 @@ def register(app):
                                 instance_data = response.json()
                                 admin_profile_ids = []
                                 for admin in instance_data['admins']:
-                                    admin_profile_ids.append(admin['person']['actor_id'].lower())
+                                    profile_id = admin['person']['actor_id']
-                                    user = find_actor_or_create(admin['person']['actor_id'])
+                                    if profile_id.startswith('https://'):
-                                    if user and not instance.user_is_admin(user.id):
+                                        admin_profile_ids.append(profile_id.lower())
-                                        new_instance_role = InstanceRole(instance_id=instance.id, user_id=user.id,
+                                        user = find_actor_or_create(profile_id)
-                                                                         role='admin')
+                                        if user and not instance.user_is_admin(user.id):
-                                        db.session.add(new_instance_role)
+                                            new_instance_role = InstanceRole(instance_id=instance.id, user_id=user.id,
+                                                                             role='admin')
+                                            db.session.add(new_instance_role)
                                 # remove any InstanceRoles that are no longer part of instance-data['admins']
                                 for instance_admin in InstanceRole.query.filter_by(instance_id=instance.id):
                                    if instance_admin.user.profile_id() not in admin_profile_ids:
@@ -294,9 +304,7 @@ def register(app):
                         finally:
                             if response:
                                 response.close()
-
+                        db.session.commit()
-                # Commit all changes at once
-                db.session.commit()
 
             except Exception as e:
                 db.session.rollback()

app/community/forms.py

@@ -155,10 +155,12 @@ class CreateImageForm(CreatePostForm):
 
     def validate(self, extra_validators=None) -> bool:
         uploaded_file = request.files['image_file']
-        if uploaded_file and uploaded_file.filename != '':
+        if uploaded_file and uploaded_file.filename != '' and not uploaded_file.filename.endswith('.svg'):
             Image.MAX_IMAGE_PIXELS = 89478485
             # Do not allow fascist meme content
             try:
+                if '.avif' in uploaded_file.filename:
+                    import pillow_avif
                 image_text = pytesseract.image_to_string(Image.open(BytesIO(uploaded_file.read())).convert('L'))
             except FileNotFoundError as e:
                 image_text = ''

app/community/routes.py

@@ -331,6 +331,7 @@ def show_community(community: Community):
                            etag=f"{community.id}{sort}{post_layout}_{hash(community.last_active)}", related_communities=related_communities,
                            next_url=next_url, prev_url=prev_url, low_bandwidth=low_bandwidth, un_moderated=un_moderated,
                            recently_upvoted=recently_upvoted, recently_downvoted=recently_downvoted,
+                           canonical=community.profile_id(),
                            rss_feed=f"https://{current_app.config['SERVER_NAME']}/community/{community.link()}/feed", rss_feed_name=f"{community.title} on {g.site.name}",
                            content_filters=content_filters, moderating_communities=moderating_communities(current_user.get_id()),
                            joined_communities=joined_communities(current_user.get_id()),
@@ -409,7 +410,7 @@ def subscribe(actor):
 # this is separated out from the subscribe route so it can be used by the 
 # admin.admin_federation.preload_form as well
 @celery.task
-def do_subscribe(actor, user_id, main_user_name=True):
+def do_subscribe(actor, user_id, admin_preload=False):
     remote = False
     actor = actor.strip()
     user = User.query.get(user_id)
@@ -423,18 +424,22 @@ def do_subscribe(actor, user_id, main_user_name=True):
     if community is not None:
         pre_load_message['community'] = community.ap_id
         if community.id in communities_banned_from(user.id):
-            if main_user_name:
+            if not admin_preload:
                 abort(401)
             else:
                 pre_load_message['user_banned'] = True
         if community_membership(user, community) != SUBSCRIPTION_MEMBER and community_membership(user, community) != SUBSCRIPTION_PENDING:
             banned = CommunityBan.query.filter_by(user_id=user.id, community_id=community.id).first()
             if banned:
-                if main_user_name:
+                if not admin_preload:
                     flash(_('You cannot join this community'))
                 else:
                     pre_load_message['community_banned_by_local_instance'] = True
             success = True
+            # for local communities, joining is instant
+            member = CommunityMember(user_id=user.id, community_id=community.id)
+            db.session.add(member)
+            db.session.commit()
             if remote:
                 # send ActivityPub message to remote community, asking to follow. Accept message will be sent to our shared inbox
                 join_request = CommunityJoinRequest(user_id=user.id, community_id=community.id)
@@ -442,47 +447,43 @@ def do_subscribe(actor, user_id, main_user_name=True):
                 db.session.commit()
                 if community.instance.online():
                     follow = {
-                      "actor": user.public_url(main_user_name=main_user_name),
+                      "actor": user.public_url(),
                       "to": [community.public_url()],
                       "object": community.public_url(),
                       "type": "Follow",
                       "id": f"https://{current_app.config['SERVER_NAME']}/activities/follow/{join_request.id}"
                     }
                     success = post_request(community.ap_inbox_url, follow, user.private_key,
-                                                           user.public_url(main_user_name=main_user_name) + '#main-key', timeout=10)
+                                                           user.public_url() + '#main-key', timeout=10)
                 if success is False or isinstance(success, str):
                     if 'is not in allowlist' in success:
                         msg_to_user = f'{community.instance.domain} does not allow us to join their communities.'
-                        if main_user_name:
+                        if not admin_preload:
                             flash(_(msg_to_user), 'error')
                         else:
                             pre_load_message['status'] = msg_to_user
                     else:
                         msg_to_user = "There was a problem while trying to communicate with remote server. If other people have already joined this community it won't matter."
-                        if main_user_name:
+                        if not admin_preload:
                             flash(_(msg_to_user), 'error')
                         else:
                             pre_load_message['status'] = msg_to_user
 
-            # for local communities, joining is instant
-            member = CommunityMember(user_id=user.id, community_id=community.id)
-            db.session.add(member)
-            db.session.commit()
             if success is True:
-                if main_user_name:
+                if not admin_preload:
                     flash('You joined ' + community.title)
                 else:
                     pre_load_message['status'] = 'joined'
         else:
-            if not main_user_name:
+            if admin_preload:
                 pre_load_message['status'] = 'already subscribed, or subsciption pending'
 
         cache.delete_memoized(community_membership, user, community)
         cache.delete_memoized(joined_communities, user.id)
-        if not main_user_name:
+        if admin_preload:
             return pre_load_message
     else:
-        if main_user_name:
+        if not admin_preload:
             abort(404)
         else:
             pre_load_message['community'] = actor
@@ -587,7 +588,7 @@ def join_then_add(actor):
 @login_required
 @validation_required
 def add_post(actor, type):
-    if current_user.banned:
+    if current_user.banned or current_user.ban_posts:
         return show_ban_message()
     community = actor_to_community(actor)
 
@@ -672,17 +673,20 @@ def add_post(actor, type):
 
                 if file_ext.lower() == '.heic':
                     register_heif_opener()
+                if file_ext.lower() == '.avif':
+                    import pillow_avif
 
                 Image.MAX_IMAGE_PIXELS = 89478485
 
                 # resize if necessary
-                img = Image.open(final_place)
+                if not final_place.endswith('.svg'):
-                if '.' + img.format.lower() in allowed_extensions:
+                    img = Image.open(final_place)
-                    img = ImageOps.exif_transpose(img)
+                    if '.' + img.format.lower() in allowed_extensions:
+                        img = ImageOps.exif_transpose(img)
 
-                    # limit full sized version to 2000px
+                        # limit full sized version to 2000px
-                    img.thumbnail((2000, 2000))
+                        img.thumbnail((2000, 2000))
-                    img.save(final_place)
+                        img.save(final_place)
 
                 request_json['object']['attachment'] = [{'type': 'Image', 'url': f'https://{current_app.config["SERVER_NAME"]}/{final_place.replace("app/", "")}',
                                                         'name': form.image_alt_text.data}]
@@ -838,7 +842,7 @@ def federate_post(community, post):
         page['oneOf' if poll.mode == 'single' else 'anyOf'] = choices
     if not community.is_local():  # this is a remote community - send the post to the instance that hosts it
         post_request_in_background(community.ap_inbox_url, create, current_user.private_key,
-                               current_user.public_url() + '#main-key')
+                               current_user.public_url() + '#main-key', timeout=10)
         flash(_('Your post to %(name)s has been made.', name=community.title))
     else:  # local community - send (announce) post out to followers
         announce = {

app/community/util.py

@@ -16,16 +16,16 @@ from app.activitypub.util import find_actor_or_create, actor_json_to_model, post
 from app.constants import POST_TYPE_ARTICLE, POST_TYPE_LINK, POST_TYPE_IMAGE, POST_TYPE_VIDEO, NOTIF_POST, \
     POST_TYPE_POLL
 from app.models import Community, File, BannedInstances, PostReply, Post, utcnow, CommunityMember, Site, \
-    Instance, Notification, User, ActivityPubLog, NotificationSubscription, PollChoice, Poll
+    Instance, Notification, User, ActivityPubLog, NotificationSubscription, PollChoice, Poll, Tag
 from app.utils import get_request, gibberish, markdown_to_html, domain_from_url, \
     is_image_url, ensure_directory_exists, shorten_string, \
     remove_tracking_from_link, ap_datetime, instance_banned, blocked_phrases, url_to_thumbnail_file, opengraph_parse, \
-    piefed_markdown_to_lemmy_markdown
+    piefed_markdown_to_lemmy_markdown, get_task_session
 from sqlalchemy import func, desc, text
 import os
 
 
-allowed_extensions = ['.gif', '.jpg', '.jpeg', '.png', '.webp', '.heic', '.mpo']
+allowed_extensions = ['.gif', '.jpg', '.jpeg', '.png', '.webp', '.heic', '.mpo', '.avif', '.svg']
 
 
 def search_for_community(address: str):
@@ -170,7 +170,7 @@ def retrieve_mods_and_backfill(community_id: int):
             return
 
         # download 50 old posts
-        if community.ap_public_url:
+        if community.ap_outbox_url:
             outbox_request = get_request(community.ap_outbox_url, headers={'Accept': 'application/activity+json'})
             if outbox_request.status_code == 200:
                 outbox_data = outbox_request.json()
@@ -426,7 +426,7 @@ def save_post(form, post: Post, type: int):
         db.session.add(post)
     else:
         db.session.execute(text('DELETE FROM "post_tag" WHERE post_id = :post_id'), {'post_id': post.id})
-    post.tags = tags_from_string(form.tags.data)
+    post.tags = tags_from_string_old(form.tags.data)
     db.session.commit()
 
     # Save poll choices. NB this will delete all votes whenever a poll is edited. Partially because it's easier to code but also to stop malicious alterations to polls after people have already voted
@@ -500,6 +500,22 @@ def tags_from_string(tags: str) -> List[dict]:
     return return_value
 
 
+def tags_from_string_old(tags: str) -> List[Tag]:
+    return_value = []
+    tags = tags.strip()
+    if tags == '':
+        return []
+    tag_list = tags.split(',')
+    tag_list = [tag.strip() for tag in tag_list]
+    for tag in tag_list:
+        if tag[0] == '#':
+            tag = tag[1:]
+        tag_to_append = find_hashtag_or_create(tag)
+        if tag_to_append:
+            return_value.append(tag_to_append)
+    return return_value
+
+
 def delete_post_from_community(post_id):
     if current_app.debug:
         delete_post_from_community_task(post_id)
@@ -633,30 +649,38 @@ def save_icon_file(icon_file, directory='communities') -> File:
 
     if file_ext.lower() == '.heic':
         register_heif_opener()
+    elif file_ext.lower() == '.avif':
+        import pillow_avif
 
     # resize if necessary
-    Image.MAX_IMAGE_PIXELS = 89478485
+    if file_ext.lower() in allowed_extensions:
-    img = Image.open(final_place)
+        if file_ext.lower() == '.svg':  # svgs don't need to be resized
-    if '.' + img.format.lower() in allowed_extensions:
+            file = File(file_path=final_place, file_name=new_filename + file_ext, alt_text=f'{directory} icon',
-        img = ImageOps.exif_transpose(img)
+                        thumbnail_path=final_place)
-        img_width = img.width
+            db.session.add(file)
-        img_height = img.height
+            return file
-        if img.width > 250 or img.height > 250:
+        else:
-            img.thumbnail((250, 250))
+            Image.MAX_IMAGE_PIXELS = 89478485
-            img.save(final_place)
+            img = Image.open(final_place)
+            img = ImageOps.exif_transpose(img)
             img_width = img.width
             img_height = img.height
-        # save a second, smaller, version as a thumbnail
+            if img.width > 250 or img.height > 250:
-        img.thumbnail((40, 40))
+                img.thumbnail((250, 250))
-        img.save(final_place_thumbnail, format="WebP", quality=93)
+                img.save(final_place)
-        thumbnail_width = img.width
+                img_width = img.width
-        thumbnail_height = img.height
+                img_height = img.height
+            # save a second, smaller, version as a thumbnail
+            img.thumbnail((40, 40))
+            img.save(final_place_thumbnail, format="WebP", quality=93)
+            thumbnail_width = img.width
+            thumbnail_height = img.height
 
-        file = File(file_path=final_place, file_name=new_filename + file_ext, alt_text=f'{directory} icon',
+            file = File(file_path=final_place, file_name=new_filename + file_ext, alt_text=f'{directory} icon',
-                    width=img_width, height=img_height, thumbnail_width=thumbnail_width,
+                        width=img_width, height=img_height, thumbnail_width=thumbnail_width,
-                    thumbnail_height=thumbnail_height, thumbnail_path=final_place_thumbnail)
+                        thumbnail_height=thumbnail_height, thumbnail_path=final_place_thumbnail)
-        db.session.add(file)
+            db.session.add(file)
-        return file
+            return file
     else:
         abort(400)
 
@@ -679,6 +703,8 @@ def save_banner_file(banner_file, directory='communities') -> File:
 
     if file_ext.lower() == '.heic':
         register_heif_opener()
+    elif file_ext.lower() == '.avif':
+        import pillow_avif
 
     # resize if necessary
     Image.MAX_IMAGE_PIXELS = 89478485
@@ -718,11 +744,12 @@ def send_to_remote_instance(instance_id: int, community_id: int, payload):
 
 @celery.task
 def send_to_remote_instance_task(instance_id: int, community_id: int, payload):
-    community = Community.query.get(community_id)
+    session = get_task_session()
+    community: Community = session.query(Community).get(community_id)
     if community:
-        instance = Instance.query.get(instance_id)
+        instance: Instance = session.query(Instance).get(instance_id)
         if instance.inbox and instance.online() and not instance_banned(instance.domain):
-            if post_request(instance.inbox, payload, community.private_key, community.ap_profile_id + '#main-key') is True:
+            if post_request(instance.inbox, payload, community.private_key, community.ap_profile_id + '#main-key', timeout=10) is True:
                 instance.last_successful_send = utcnow()
                 instance.failures = 0
             else:
@@ -731,7 +758,8 @@ def send_to_remote_instance_task(instance_id: int, community_id: int, payload):
                 instance.start_trying_again = utcnow() + timedelta(seconds=instance.failures ** 4)
                 if instance.failures > 10:
                     instance.dormant = True
-            db.session.commit()
+            session.commit()
+    session.close()
 
 
 def community_in_list(community_id, community_list):

app/constants.py

@@ -36,3 +36,37 @@ ROLE_STAFF = 3
 ROLE_ADMIN = 4
 
 MICROBLOG_APPS = ["mastodon", "misskey", "akkoma", "iceshrimp", "pleroma"]
+
+APLOG_IN            = True
+
+APLOG_MONITOR       = (True, 'Debug this')
+
+APLOG_SUCCESS       = (True, 'success')
+APLOG_FAILURE       = (True, 'failure')
+APLOG_IGNORED       = (True, 'ignored')
+APLOG_PROCESSING    = (True, 'processing')
+
+APLOG_NOTYPE        = (True, 'Unknown')
+APLOG_DUPLICATE     = (True, 'Duplicate')
+APLOG_FOLLOW        = (True, 'Follow')
+APLOG_ACCEPT        = (True, 'Accept')
+APLOG_DELETE        = (True, 'Delete')
+APLOG_CHATMESSAGE   = (True, 'Create ChatMessage')
+APLOG_CREATE        = (True, 'Create')
+APLOG_UPDATE        = (True, 'Update')
+APLOG_LIKE          = (True, 'Like')
+APLOG_DISLIKE       = (True, 'Dislike')
+APLOG_REPORT        = (True, 'Report')
+APLOG_USERBAN       = (True, 'User Ban')
+APLOG_LOCK          = (True, 'Post Lock')
+
+APLOG_UNDO_FOLLOW   = (True, 'Undo Follow')
+APLOG_UNDO_DELETE   = (True, 'Undo Delete')
+APLOG_UNDO_VOTE     = (True, 'Undo Vote')
+APLOG_UNDO_USERBAN  = (True, 'Undo User Ban')
+
+APLOG_ADD           = (True, 'Add Mod/Sticky')
+APLOG_REMOVE        = (True, 'Remove Mod/Sticky')
+
+APLOG_ANNOUNCE      = (True, 'Announce')
+APLOG_PT_VIEW       = (True, 'PeerTube View')

app/main/routes.py

@@ -9,8 +9,7 @@ from sqlalchemy.sql.operators import or_, and_
 from app import db, cache
 from app.activitypub.util import users_total, active_month, local_posts, local_communities
 from app.activitypub.signature import default_context, LDSignature
-from app.constants import SUBSCRIPTION_PENDING, SUBSCRIPTION_MEMBER, POST_TYPE_IMAGE, POST_TYPE_LINK, \
+from app.constants import SUBSCRIPTION_PENDING, SUBSCRIPTION_MEMBER, SUBSCRIPTION_OWNER, SUBSCRIPTION_MODERATOR
-    SUBSCRIPTION_OWNER, SUBSCRIPTION_MODERATOR, POST_TYPE_VIDEO, POST_TYPE_POLL
 from app.email import send_email
 from app.inoculation import inoculation
 from app.main import bp
@@ -22,7 +21,7 @@ from app.utils import render_template, get_setting, request_etag_matches, return
     ap_datetime, shorten_string, markdown_to_text, user_filters_home, \
     joined_communities, moderating_communities, markdown_to_html, allowlist_html, \
     blocked_instances, communities_banned_from, topic_tree, recently_upvoted_posts, recently_downvoted_posts, \
-    blocked_users, menu_topics, languages_for_form, blocked_communities, get_request
+    blocked_users, menu_topics, blocked_communities, get_request
 from app.models import Community, CommunityMember, Post, Site, User, utcnow, Topic, Instance, \
     Notification, Language, community_language, ModLog, read_posts
 
@@ -61,7 +60,7 @@ def home_page(sort, view_filter):
     if current_user.is_anonymous:
         flash(_('Create an account to tailor this feed to your interests.'))
         posts = Post.query.filter(Post.from_bot == False, Post.nsfw == False, Post.nsfl == False, Post.deleted == False)
-        content_filters = {}
+        content_filters = {'trump': {'trump', 'elon', 'musk'}}
     else:
         posts = Post.query.filter(Post.deleted == False)
 
@@ -91,7 +90,7 @@ def home_page(sort, view_filter):
         content_filters = user_filters_home(current_user.id)
 
     # view filter - subscribed/local/all
-    if view_filter == 'subscribed':
+    if view_filter == 'subscribed' and current_user.is_authenticated:
         posts = posts.join(CommunityMember, Post.community_id == CommunityMember.community_id).filter(CommunityMember.is_banned == False)
         posts = posts.filter(CommunityMember.user_id == current_user.id)
     elif view_filter == 'local':
@@ -100,7 +99,9 @@ def home_page(sort, view_filter):
     elif view_filter == 'popular':
         posts = posts.join(Community, Community.id == Post.community_id)
         posts = posts.filter(Community.show_popular == True, Post.score > 100)
-    elif view_filter == 'all':
+        if current_user.is_anonymous:
+            posts = posts.filter(Community.low_quality == False)
+    elif view_filter == 'all' or current_user.is_anonymous:
         posts = posts.join(Community, Community.id == Post.community_id)
         posts = posts.filter(Community.show_all == True)
 
@@ -216,7 +217,7 @@ def list_communities():
     return render_template('list_communities.html', communities=communities, search=search_param, title=_('Communities'),
                            SUBSCRIPTION_PENDING=SUBSCRIPTION_PENDING, SUBSCRIPTION_MEMBER=SUBSCRIPTION_MEMBER,
                            SUBSCRIPTION_OWNER=SUBSCRIPTION_OWNER, SUBSCRIPTION_MODERATOR=SUBSCRIPTION_MODERATOR,
-                           next_url=next_url, prev_url=prev_url,
+                           next_url=next_url, prev_url=prev_url, current_user=current_user,
                            topics=topics, languages=languages, topic_id=topic_id, language_id=language_id, sort_by=sort_by,
                            low_bandwidth=low_bandwidth, moderating_communities=moderating_communities(current_user.get_id()),
                            joined_communities=joined_communities(current_user.get_id()),
@@ -262,13 +263,13 @@ def list_local_communities():
     # Pagination
     communities = communities.paginate(page=page, per_page=250 if current_user.is_authenticated and not low_bandwidth else 50,
                            error_out=False)
-    next_url = url_for('main.list_communities', page=communities.next_num, sort_by=sort_by, language_id=language_id) if communities.has_next else None
+    next_url = url_for('main.list_local_communities', page=communities.next_num, sort_by=sort_by, language_id=language_id) if communities.has_next else None
-    prev_url = url_for('main.list_communities', page=communities.prev_num, sort_by=sort_by, language_id=language_id) if communities.has_prev and page != 1 else None
+    prev_url = url_for('main.list_local_communities', page=communities.prev_num, sort_by=sort_by, language_id=language_id) if communities.has_prev and page != 1 else None
 
     return render_template('list_communities.html', communities=communities, search=search_param, title=_('Local Communities'),
                            SUBSCRIPTION_PENDING=SUBSCRIPTION_PENDING, SUBSCRIPTION_MEMBER=SUBSCRIPTION_MEMBER,
                            SUBSCRIPTION_OWNER=SUBSCRIPTION_OWNER, SUBSCRIPTION_MODERATOR=SUBSCRIPTION_MODERATOR,
-                           next_url=next_url, prev_url=prev_url,
+                           next_url=next_url, prev_url=prev_url, current_user=current_user,
                            topics=topics, languages=languages, topic_id=topic_id, language_id=language_id, sort_by=sort_by,
                            low_bandwidth=low_bandwidth, moderating_communities=moderating_communities(current_user.get_id()),
                            joined_communities=joined_communities(current_user.get_id()),
@@ -309,8 +310,8 @@ def list_subscribed_communities():
         # Pagination
         communities = communities.paginate(page=page, per_page=250 if current_user.is_authenticated and not low_bandwidth else 50,
                            error_out=False)
-        next_url = url_for('main.list_communities', page=communities.next_num, sort_by=sort_by, language_id=language_id) if communities.has_next else None
+        next_url = url_for('main.list_subscribed_communities', page=communities.next_num, sort_by=sort_by, language_id=language_id) if communities.has_next else None
-        prev_url = url_for('main.list_communities', page=communities.prev_num, sort_by=sort_by, language_id=language_id) if communities.has_prev and page != 1 else None
+        prev_url = url_for('main.list_subscribed_communities', page=communities.prev_num, sort_by=sort_by, language_id=language_id) if communities.has_prev and page != 1 else None
 
     else:
         communities = []
@@ -320,13 +321,76 @@ def list_subscribed_communities():
     return render_template('list_communities.html', communities=communities, search=search_param, title=_('Joined Communities'),
                            SUBSCRIPTION_PENDING=SUBSCRIPTION_PENDING, SUBSCRIPTION_MEMBER=SUBSCRIPTION_MEMBER,
                            SUBSCRIPTION_OWNER=SUBSCRIPTION_OWNER, SUBSCRIPTION_MODERATOR=SUBSCRIPTION_MODERATOR,
-                           next_url=next_url, prev_url=prev_url,
+                           next_url=next_url, prev_url=prev_url, current_user=current_user,
                            topics=topics, languages=languages, topic_id=topic_id, language_id=language_id, sort_by=sort_by,
                            low_bandwidth=low_bandwidth, moderating_communities=moderating_communities(current_user.get_id()),
                            joined_communities=joined_communities(current_user.get_id()),
                            menu_topics=menu_topics(), site=g.site)
 
 
+@bp.route('/communities/notsubscribed', methods=['GET'])
+def list_not_subscribed_communities():
+    verification_warning()
+    search_param = request.args.get('search', '')
+    topic_id = int(request.args.get('topic_id', 0))
+    language_id = int(request.args.get('language_id', 0))
+    page = request.args.get('page', 1, type=int)
+    low_bandwidth = request.cookies.get('low_bandwidth', '0') == '1'
+    sort_by = request.args.get('sort_by', 'post_reply_count desc')
+    topics = Topic.query.order_by(Topic.name).all()
+    languages = Language.query.order_by(Language.name).all()
+    if current_user.is_authenticated:
+        # get all communities
+        all_communities = Community.query.filter_by(banned=False)
+        # get the user's joined communities
+        joined_communities = Community.query.filter_by(banned=False).join(CommunityMember).filter(CommunityMember.user_id == current_user.id)
+        # get the joined community ids list
+        joined_ids = []
+        for jc in joined_communities:
+            joined_ids.append(jc.id)
+        # filter out the joined communities from all communities
+        communities = all_communities.filter(Community.id.not_in(joined_ids))
+
+        if search_param == '':
+            pass
+        else:
+            communities = communities.filter(or_(Community.title.ilike(f"%{search_param}%"), Community.ap_id.ilike(f"%{search_param}%")))
+
+        if topic_id != 0:
+            communities = communities.filter_by(topic_id=topic_id)
+
+        if language_id != 0:
+            communities = communities.join(community_language).filter(community_language.c.language_id == language_id)
+
+        banned_from = communities_banned_from(current_user.id)
+        if banned_from:
+            communities = communities.filter(Community.id.not_in(banned_from))
+        if current_user.hide_nsfw == 1:
+            communities = communities.filter(Community.nsfw == False)
+        if current_user.hide_nsfl == 1:
+            communities = communities.filter(Community.nsfl == False)
+
+        communities = communities.order_by(text('community.' + sort_by))
+
+        # Pagination
+        communities = communities.paginate(page=page, per_page=250 if current_user.is_authenticated and not low_bandwidth else 50,
+                           error_out=False)
+        next_url = url_for('main.list_not_subscribed_communities', page=communities.next_num, sort_by=sort_by, language_id=language_id) if communities.has_next else None
+        prev_url = url_for('main.list_not_subscribed_communities', page=communities.prev_num, sort_by=sort_by, language_id=language_id) if communities.has_prev and page != 1 else None
+
+    else:
+        communities = []
+        next_url = None
+        prev_url = None
+
+    return render_template('list_communities.html', communities=communities, search=search_param, title=_('Not Joined Communities'),
+                           SUBSCRIPTION_PENDING=SUBSCRIPTION_PENDING, SUBSCRIPTION_MEMBER=SUBSCRIPTION_MEMBER,
+                           SUBSCRIPTION_OWNER=SUBSCRIPTION_OWNER, SUBSCRIPTION_MODERATOR=SUBSCRIPTION_MODERATOR,
+                           next_url=next_url, prev_url=prev_url, current_user=current_user,
+                           topics=topics, languages=languages, topic_id=topic_id, language_id=language_id, sort_by=sort_by,
+                           low_bandwidth=low_bandwidth, moderating_communities=moderating_communities(current_user.get_id()),
+                           menu_topics=menu_topics(), site=g.site)
+
 @bp.route('/modlog', methods=['GET'])
 def modlog():
     page = request.args.get('page', 1, type=int)
@@ -426,6 +490,27 @@ def list_files(directory):
             yield os.path.join(root, file)
 
 
+@bp.route('/replay_inbox')
+@login_required
+def replay_inbox():
+    from app.activitypub.routes import replay_inbox_request
+
+    request_json = {}
+    """
+    request_json = {"@context": ["https://join-lemmy.org/context.json", "https://www.w3.org/ns/activitystreams"],
+                    "actor": "https://lemmy.lemmy/u/doesnotexist",
+                    "cc": [],
+                    "id": "https://lemmy.lemmy/activities/delete/5d42c8bf-cc60-4d2c-a3b5-673ddb7ce64b",
+                    "object": "https://lemmy.lemmy/u/doesnotexist",
+                    "to": ["https://www.w3.org/ns/activitystreams#Public"],
+                    "type": "Delete"}
+    """
+
+    replay_inbox_request(request_json)
+
+    return 'ok'
+
+
 @bp.route('/test')
 def test():
 

app/models.py

@@ -8,6 +8,7 @@ import arrow
 from flask import current_app, escape, url_for, render_template_string
 from flask_login import UserMixin, current_user
 from sqlalchemy import or_, text, desc
+from sqlalchemy.exc import IntegrityError
 from werkzeug.security import generate_password_hash, check_password_hash
 from flask_babel import _, lazy_gettext as _l
 from sqlalchemy.orm import backref
@@ -51,7 +52,7 @@ class AllowedInstances(db.Model):
 
 class Instance(db.Model):
     id = db.Column(db.Integer, primary_key=True)
-    domain = db.Column(db.String(256), index=True)
+    domain = db.Column(db.String(256), index=True, unique=True)
     inbox = db.Column(db.String(256))
     shared_inbox = db.Column(db.String(256))
     outbox = db.Column(db.String(256))
@@ -134,12 +135,20 @@ class InstanceRole(db.Model):
     user = db.relationship('User', lazy='joined')
 
 
+# Instances that this user has blocked
 class InstanceBlock(db.Model):
     user_id = db.Column(db.Integer, db.ForeignKey('user.id'), primary_key=True)
     instance_id = db.Column(db.Integer, db.ForeignKey('instance.id'), primary_key=True)
     created_at = db.Column(db.DateTime, default=utcnow)
 
 
+# Instances that have banned this user
+class InstanceBan(db.Model):
+    user_id = db.Column(db.Integer, db.ForeignKey('user.id'), primary_key=True)
+    instance_id = db.Column(db.Integer, db.ForeignKey('instance.id'), primary_key=True)
+    banned_until = db.Column(db.DateTime)
+
+
 class Conversation(db.Model):
     id = db.Column(db.Integer, primary_key=True)
     user_id = db.Column(db.Integer, db.ForeignKey('user.id'), index=True)
@@ -225,6 +234,11 @@ class Tag(db.Model):
     banned = db.Column(db.Boolean, default=False, index=True)
 
 
+class Licence(db.Model):
+    id = db.Column(db.Integer, primary_key=True)
+    name = db.Column(db.String(50))
+
+
 class Language(db.Model):
     id = db.Column(db.Integer, primary_key=True)
     code = db.Column(db.String(5), index=True)
@@ -262,7 +276,8 @@ class File(db.Model):
                 return self.source_url
         elif self.file_path:
             file_path = self.file_path[4:] if self.file_path.startswith('app/') else self.file_path
-            return f"https://{current_app.config['SERVER_NAME']}/{file_path}"
+            scheme = 'http' if current_app.config['SERVER_NAME'] == '127.0.0.1:5000' else 'https'
+            return f"{scheme}://{current_app.config['SERVER_NAME']}/{file_path}"
         else:
             return ''
 
@@ -270,7 +285,8 @@ class File(db.Model):
         if self.file_path is None:
             return self.thumbnail_url()
         file_path = self.file_path[4:] if self.file_path.startswith('app/') else self.file_path
-        return f"https://{current_app.config['SERVER_NAME']}/{file_path}"
+        scheme = 'http' if current_app.config['SERVER_NAME'] == '127.0.0.1:5000' else 'https'
+        return f"{scheme}://{current_app.config['SERVER_NAME']}/{file_path}"
 
     def thumbnail_url(self):
         if self.thumbnail_path is None:
@@ -279,7 +295,8 @@ class File(db.Model):
             else:
                 return ''
         thumbnail_path = self.thumbnail_path[4:] if self.thumbnail_path.startswith('app/') else self.thumbnail_path
-        return f"https://{current_app.config['SERVER_NAME']}/{thumbnail_path}"
+        scheme = 'http' if current_app.config['SERVER_NAME'] == '127.0.0.1:5000' else 'https'
+        return f"{scheme}://{current_app.config['SERVER_NAME']}/{thumbnail_path}"
 
     def delete_from_disk(self):
         purge_from_cache = []
@@ -366,6 +383,7 @@ class Topic(db.Model):
     name = db.Column(db.String(50))
     num_communities = db.Column(db.Integer, default=0)
     parent_id = db.Column(db.Integer)
+    show_posts_in_children = db.Column(db.Boolean, default=False)
     communities = db.relationship('Community', lazy='dynamic', backref='topic', cascade="all, delete-orphan")
 
     def path(self):
@@ -417,7 +435,7 @@ class Community(db.Model):
     posting_warning = db.Column(db.String(512))
 
     ap_id = db.Column(db.String(255), index=True)
-    ap_profile_id = db.Column(db.String(255), index=True)
+    ap_profile_id = db.Column(db.String(255), index=True, unique=True)
     ap_followers_url = db.Column(db.String(255))
     ap_preferred_username = db.Column(db.String(255))
     ap_discoverable = db.Column(db.Boolean, default=False)
@@ -438,7 +456,6 @@ class Community(db.Model):
     private_mods = db.Column(db.Boolean, default=False)
 
     # Which feeds posts from this community show up in
-    show_home = db.Column(db.Boolean, default=False)        # For anonymous users. When logged in, the home feed shows posts from subscribed communities
     show_popular = db.Column(db.Boolean, default=True)
     show_all = db.Column(db.Boolean, default=True)
 
@@ -613,6 +630,7 @@ class Community(db.Model):
         db.session.query(CommunityJoinRequest).filter(CommunityJoinRequest.community_id == self.id).delete()
         db.session.query(CommunityMember).filter(CommunityMember.community_id == self.id).delete()
         db.session.query(Report).filter(Report.suspect_community_id == self.id).delete()
+        db.session.query(ModLog).filter(ModLog.community_id == self.id).delete()
 
 
 user_role = db.Table('user_role',
@@ -639,7 +657,10 @@ class User(UserMixin, db.Model):
     password_hash = db.Column(db.String(128))
     verified = db.Column(db.Boolean, default=False)
     verification_token = db.Column(db.String(16), index=True)
-    banned = db.Column(db.Boolean, default=False)
+    banned = db.Column(db.Boolean, default=False, index=True)
+    banned_until = db.Column(db.DateTime)   # null == permanent ban
+    ban_posts = db.Column(db.Boolean, default=False)
+    ban_comments = db.Column(db.Boolean, default=False)
     deleted = db.Column(db.Boolean, default=False)
     deleted_by = db.Column(db.Integer, index=True)
     about = db.Column(db.Text)      # markdown
@@ -691,7 +712,7 @@ class User(UserMixin, db.Model):
     conversations = db.relationship('Conversation', lazy='dynamic', secondary=conversation_member, backref=db.backref('members', lazy='joined'))
 
     ap_id = db.Column(db.String(255), index=True)           # e.g. username@server
-    ap_profile_id = db.Column(db.String(255), index=True)   # e.g. https://server/u/username
+    ap_profile_id = db.Column(db.String(255), index=True, unique=True)   # e.g. https://server/u/username
     ap_public_url = db.Column(db.String(255))               # e.g. https://server/u/UserName
     ap_fetched_at = db.Column(db.DateTime)
     ap_followers_url = db.Column(db.String(255))
@@ -890,20 +911,21 @@ class User(UserMixin, db.Model):
 
     def recalculate_attitude(self):
         upvotes = downvotes = 0
-        last_50_votes = PostVote.query.filter(PostVote.user_id == self.id).order_by(-PostVote.id).limit(50)
+        with db.session.no_autoflush:  # Avoid StaleDataError exception
-        for vote in last_50_votes:
+            last_50_votes = PostVote.query.filter(PostVote.user_id == self.id).order_by(-PostVote.id).limit(50)
-            if vote.effect > 0:
+            for vote in last_50_votes:
-                upvotes += 1
+                if vote.effect > 0:
-            if vote.effect < 0:
+                    upvotes += 1
-                downvotes += 1
+                if vote.effect < 0:
+                    downvotes += 1
 
-        comment_upvotes = comment_downvotes = 0
+            comment_upvotes = comment_downvotes = 0
-        last_50_votes = PostReplyVote.query.filter(PostReplyVote.user_id == self.id).order_by(-PostReplyVote.id).limit(50)
+            last_50_votes = PostReplyVote.query.filter(PostReplyVote.user_id == self.id).order_by(-PostReplyVote.id).limit(50)
-        for vote in last_50_votes:
+            for vote in last_50_votes:
-            if vote.effect > 0:
+                if vote.effect > 0:
-                comment_upvotes += 1
+                    comment_upvotes += 1
-            if vote.effect < 0:
+                if vote.effect < 0:
-                comment_downvotes += 1
+                    comment_downvotes += 1
 
         total_upvotes = upvotes + comment_upvotes
         total_downvotes = downvotes + comment_downvotes
@@ -999,6 +1021,7 @@ class User(UserMixin, db.Model):
         db.session.query(PollChoiceVote).filter(PollChoiceVote.user_id == self.id).delete()
         db.session.query(PostBookmark).filter(PostBookmark.user_id == self.id).delete()
         db.session.query(PostReplyBookmark).filter(PostReplyBookmark.user_id == self.id).delete()
+        db.session.query(ModLog).filter(ModLog.user_id == self.id).delete()
 
     def purge_content(self, soft=True):
         files = File.query.join(Post).filter(Post.user_id == self.id).all()
@@ -1073,6 +1096,7 @@ class Post(db.Model):
     image_id = db.Column(db.Integer, db.ForeignKey('file.id'), index=True)
     domain_id = db.Column(db.Integer, db.ForeignKey('domain.id'), index=True)
     instance_id = db.Column(db.Integer, db.ForeignKey('instance.id'), index=True)
+    licence_id = db.Column(db.Integer, db.ForeignKey('licence.id'), index=True)
     slug = db.Column(db.String(255))
     title = db.Column(db.String(255))
     url = db.Column(db.String(2048))
@@ -1106,7 +1130,7 @@ class Post(db.Model):
     cross_posts = db.Column(MutableList.as_mutable(ARRAY(db.Integer)))
     tags = db.relationship('Tag', lazy='dynamic', secondary=post_tag, backref=db.backref('posts', lazy='dynamic'))
 
-    ap_id = db.Column(db.String(255), index=True)
+    ap_id = db.Column(db.String(255), index=True, unique=True)
     ap_create_id = db.Column(db.String(100))
     ap_announce_id = db.Column(db.String(100))
 
@@ -1118,6 +1142,7 @@ class Post(db.Model):
     community = db.relationship('Community', lazy='joined', overlaps='posts', foreign_keys=[community_id])
     replies = db.relationship('PostReply', lazy='dynamic', backref='post')
     language = db.relationship('Language', foreign_keys=[language_id])
+    licence = db.relationship('Licence', foreign_keys=[licence_id])
 
     # db relationship tracked by the "read_posts" table
     # this is the Post side, so its referencing the User side
@@ -1129,12 +1154,12 @@ class Post(db.Model):
 
     @classmethod
     def get_by_ap_id(cls, ap_id):
-        return cls.query.filter_by(ap_id=ap_id).first()
+        return cls.query.filter_by(ap_id=ap_id.lower()).first()
 
     @classmethod
     def new(cls, user: User, community: Community, request_json: dict, announce_id=None):
         from app.activitypub.util import instance_weight, find_language_or_create, find_language, find_hashtag_or_create, \
-            make_image_sizes, notify_about_post
+            find_licence_or_create, make_image_sizes, notify_about_post
         from app.utils import allowlist_html, markdown_to_html, html_to_text, microblog_content_to_title, blocked_phrases, \
             is_image_url, is_video_url, domain_from_url, opengraph_parse, shorten_string, remove_tracking_from_link, \
             is_video_hosting_site, communities_banned_from
@@ -1155,7 +1180,7 @@ class Post(db.Model):
                     sticky=request_json['object']['stickied'] if 'stickied' in request_json['object'] else False,
                     nsfw=request_json['object']['sensitive'] if 'sensitive' in request_json['object'] else False,
                     nsfl=request_json['object']['nsfl'] if 'nsfl' in request_json['object'] else nsfl_in_title,
-                    ap_id=request_json['object']['id'],
+                    ap_id=request_json['object']['id'].lower(),
                     ap_create_id=request_json['id'],
                     ap_announce_id=announce_id,
                     up_votes=1,
@@ -1205,8 +1230,10 @@ class Post(db.Model):
                 if blocked_phrase in post.body:
                     return None
 
-        if 'attachment' in request_json['object'] and len(request_json['object']['attachment']) > 0 and \
+        if ('attachment' in request_json['object'] and
-                'type' in request_json['object']['attachment'][0]:
+            isinstance(request_json['object']['attachment'], list) and
+            len(request_json['object']['attachment']) > 0 and
+            'type' in request_json['object']['attachment'][0]):
             alt_text = None
             if request_json['object']['attachment'][0]['type'] == 'Link':
                 post.url = request_json['object']['attachment'][0]['href']  # Lemmy < 0.19.4
@@ -1218,46 +1245,46 @@ class Post(db.Model):
                 post.url = request_json['object']['attachment'][0]['url']  # PixelFed, PieFed, Lemmy >= 0.19.4
                 if 'name' in request_json['object']['attachment'][0]:
                     alt_text = request_json['object']['attachment'][0]['name']
-            if post.url:
+
-                if is_image_url(post.url):
+        if 'attachment' in request_json['object'] and isinstance(request_json['object']['attachment'], dict):  # a.gup.pe (Mastodon)
-                    post.type = constants.POST_TYPE_IMAGE
+            alt_text = None
-                    if 'image' in request_json['object'] and 'url' in request_json['object']['image']:
+            post.url = request_json['object']['attachment']['url']
-                        image = File(source_url=request_json['object']['image']['url'])
+
-                    else:
+        if post.url:
-                        image = File(source_url=post.url)
+            if is_image_url(post.url):
-                    if alt_text:
+                post.type = constants.POST_TYPE_IMAGE
-                        image.alt_text = alt_text
+                image = File(source_url=post.url)
-                    db.session.add(image)
+                if alt_text:
-                    post.image = image
+                    image.alt_text = alt_text
-                elif is_video_url(post.url):  # youtube is detected later
+                db.session.add(image)
-                    post.type = constants.POST_TYPE_VIDEO
+                post.image = image
-                    image = File(source_url=post.url)
+            elif is_video_url(post.url):  # youtube is detected later
-                    db.session.add(image)
+                post.type = constants.POST_TYPE_VIDEO
-                    post.image = image
+                # custom thumbnails will be added below in the "if 'image' in request_json['object'] and post.image is None:" section
-                else:
+            else:
-                    post.type = constants.POST_TYPE_LINK
+                post.type = constants.POST_TYPE_LINK
-                domain = domain_from_url(post.url)
+            domain = domain_from_url(post.url)
-                # notify about links to banned websites.
+            # notify about links to banned websites.
-                already_notified = set()  # often admins and mods are the same people - avoid notifying them twice
+            already_notified = set()  # often admins and mods are the same people - avoid notifying them twice
-                if domain.notify_mods:
+            if domain.notify_mods:
-                    for community_member in post.community.moderators():
+                for community_member in post.community.moderators():
-                        notify = Notification(title='Suspicious content', url=post.ap_id,
+                    notify = Notification(title='Suspicious content', url=post.ap_id,
-                                              user_id=community_member.user_id,
+                                          user_id=community_member.user_id,
+                                          author_id=user.id)
+                    db.session.add(notify)
+                    already_notified.add(community_member.user_id)
+            if domain.notify_admins:
+                for admin in Site.admins():
+                    if admin.id not in already_notified:
+                        notify = Notification(title='Suspicious content',
+                                              url=post.ap_id, user_id=admin.id,
                                               author_id=user.id)
                         db.session.add(notify)
-                        already_notified.add(community_member.user_id)
+            if domain.banned or domain.name.endswith('.pages.dev'):
-                if domain.notify_admins:
+                raise Exception(domain.name + ' is blocked by admin')
-                    for admin in Site.admins():
+            else:
-                        if admin.id not in already_notified:
+                domain.post_count += 1
-                            notify = Notification(title='Suspicious content',
+                post.domain = domain
-                                                  url=post.ap_id, user_id=admin.id,
-                                                  author_id=user.id)
-                            db.session.add(notify)
-                if domain.banned or domain.name.endswith('.pages.dev'):
-                    raise Exception(domain.name + ' is blocked by admin')
-                else:
-                    domain.post_count += 1
-                    post.domain = domain
 
         if post is not None:
             if request_json['object']['type'] == 'Video':
@@ -1272,10 +1299,13 @@ class Post(db.Model):
             if 'language' in request_json['object'] and isinstance(request_json['object']['language'], dict):
                 language = find_language_or_create(request_json['object']['language']['identifier'],
                                                    request_json['object']['language']['name'])
-                post.language_id = language.id
+                post.language = language
             elif 'contentMap' in request_json['object'] and isinstance(request_json['object']['contentMap'], dict):
                 language = find_language(next(iter(request_json['object']['contentMap'])))
                 post.language_id = language.id if language else None
+            if 'licence' in request_json['object'] and isinstance(request_json['object']['licence'], dict):
+                licence = find_licence_or_create(request_json['object']['licence']['name'])
+                post.licence = licence
             if 'tag' in request_json['object'] and isinstance(request_json['object']['tag'], list):
                 for json_tag in request_json['object']['tag']:
                     if json_tag and json_tag['type'] == 'Hashtag':
@@ -1313,7 +1343,11 @@ class Post(db.Model):
             community.post_count += 1
             community.last_active = utcnow()
             user.post_count += 1
-            db.session.commit()
+            try:
+                db.session.commit()
+            except IntegrityError:
+                db.session.rollback()
+                return Post.query.filter_by(ap_id=request_json['object']['id'].lower()).one()
 
             # Polls need to be processed quite late because they need a post_id to refer to
             if request_json['object']['type'] == 'Question':
@@ -1614,7 +1648,7 @@ class PostReply(db.Model):
     edited_at = db.Column(db.DateTime)
     reports = db.Column(db.Integer, default=0)  # how many times this post has been reported. Set to -1 to ignore reports
 
-    ap_id = db.Column(db.String(255), index=True)
+    ap_id = db.Column(db.String(255), index=True, unique=True)
     ap_create_id = db.Column(db.String(100))
     ap_announce_id = db.Column(db.String(100))
 
@@ -1633,6 +1667,9 @@ class PostReply(db.Model):
         if not post.comments_enabled:
             raise Exception('Comments are disabled on this post')
 
+        if user.ban_comments:
+            raise Exception('Banned from commenting')
+
         if in_reply_to is not None:
             parent_id = in_reply_to.id
             depth = in_reply_to.depth + 1
@@ -1647,7 +1684,7 @@ class PostReply(db.Model):
                           from_bot=user.bot, nsfw=post.nsfw, nsfl=post.nsfl,
                           notify_author=notify_author, instance_id=user.instance_id,
                           language_id=language_id,
-                          ap_id=request_json['object']['id'] if request_json else None,
+                          ap_id=request_json['object']['id'].lower() if request_json else None,
                           ap_create_id=request_json['id'] if request_json else None,
                           ap_announce_id=announce_id)
         if reply.body:
@@ -1672,8 +1709,12 @@ class PostReply(db.Model):
         if reply_is_stupid(reply.body):
             raise Exception('Low quality reply')
 
-        db.session.add(reply)
+        try:
-        db.session.commit()
+            db.session.add(reply)
+            db.session.commit()
+        except IntegrityError:
+            db.session.rollback()
+            return PostReply.query.filter_by(ap_id=request_json['object']['id'].lower()).one()
 
         # Notify subscribers
         notify_about_post_reply(in_reply_to, reply)
@@ -1729,7 +1770,7 @@ class PostReply(db.Model):
 
     @classmethod
     def get_by_ap_id(cls, ap_id):
-        return cls.query.filter_by(ap_id=ap_id).first()
+        return cls.query.filter_by(ap_id=ap_id.lower()).first()
 
     def profile_id(self):
         if self.ap_id:
@@ -1880,6 +1921,7 @@ class PostReply(db.Model):
                                  effect=effect)
             self.author.reputation += effect
             db.session.add(vote)
+        db.session.commit()
         user.last_seen = utcnow()
         self.ranking = PostReply.confidence(self.up_votes, self.down_votes)
         user.recalculate_attitude()
@@ -1887,7 +1929,6 @@ class PostReply(db.Model):
         return undo
 
 
-
 class Domain(db.Model):
     id = db.Column(db.Integer, primary_key=True)
     name = db.Column(db.String(255), index=True)
@@ -1895,6 +1936,7 @@ class Domain(db.Model):
     banned = db.Column(db.Boolean, default=False, index=True) # Domains can be banned site-wide (by admin) or DomainBlock'ed by users
     notify_mods = db.Column(db.Boolean, default=False, index=True)
     notify_admins = db.Column(db.Boolean, default=False, index=True)
+    post_warning = db.Column(db.String(512))
 
     def blocked_by(self, user):
         block = DomainBlock.query.filter_by(domain_id=self.id, user_id=user.id).first()
@@ -1988,7 +2030,8 @@ class CommunityBan(db.Model):
     user_id = db.Column(db.Integer, db.ForeignKey('user.id'), primary_key=True)             # person who is banned, not the banner
     community_id = db.Column(db.Integer, db.ForeignKey('community.id'), primary_key=True)
     banned_by = db.Column(db.Integer, db.ForeignKey('user.id'))
-    reason = db.Column(db.String(50))
+    banned_until = db.Column(db.DateTime)
+    reason = db.Column(db.String(256))
     created_at = db.Column(db.DateTime, default=utcnow)
     ban_until = db.Column(db.DateTime)
 
@@ -2045,7 +2088,7 @@ class UserRegistration(db.Model):
 class PostVote(db.Model):
     id = db.Column(db.Integer, primary_key=True)
     user_id = db.Column(db.Integer, db.ForeignKey('user.id'), index=True)
-    author_id = db.Column(db.Integer, db.ForeignKey('user.id'))
+    author_id = db.Column(db.Integer, db.ForeignKey('user.id'), index=True)
     post_id = db.Column(db.Integer, db.ForeignKey('post.id'), index=True)
     effect = db.Column(db.Float, index=True)
     created_at = db.Column(db.DateTime, default=utcnow)
@@ -2055,7 +2098,7 @@ class PostVote(db.Model):
 class PostReplyVote(db.Model):
     id = db.Column(db.Integer, primary_key=True)
     user_id = db.Column(db.Integer, db.ForeignKey('user.id'), index=True)   # who voted
-    author_id = db.Column(db.Integer, db.ForeignKey('user.id')) # the author of the reply voted on - who's reputation is affected
+    author_id = db.Column(db.Integer, db.ForeignKey('user.id'), index=True) # the author of the reply voted on - who's reputation is affected
     post_reply_id = db.Column(db.Integer, db.ForeignKey('post_reply.id'), index=True)
     effect = db.Column(db.Float)
     created_at = db.Column(db.DateTime, default=utcnow)
@@ -2239,6 +2282,8 @@ class ModLog(db.Model):
         'undelete_user': _l('Restored account'),
         'ban_user': _l('Banned account'),
         'unban_user': _l('Un-banned account'),
+        'lock_post': _l('Lock post'),
+        'unlock_post': _l('Un-lock post'),
     }
 
     def action_to_str(self):

app/post/routes.py

@@ -10,7 +10,7 @@ from wtforms import SelectField, RadioField
 
 from app import db, constants, cache, celery
 from app.activitypub.signature import HttpSignature, post_request, default_context, post_request_in_background
-from app.activitypub.util import notify_about_post_reply, inform_followers_of_post_update
+from app.activitypub.util import notify_about_post_reply, inform_followers_of_post_update, update_post_from_activity
 from app.community.util import save_post, send_to_remote_instance
 from app.inoculation import inoculation
 from app.post.forms import NewReplyForm, ReportPostForm, MeaCulpaForm, CrossPostForm
@@ -23,7 +23,7 @@ from app.constants import SUBSCRIPTION_MEMBER, SUBSCRIPTION_OWNER, SUBSCRIPTION_
 from app.models import Post, PostReply, \
     PostReplyVote, PostVote, Notification, utcnow, UserBlock, DomainBlock, InstanceBlock, Report, Site, Community, \
     Topic, User, Instance, NotificationSubscription, UserFollower, Poll, PollChoice, PollChoiceVote, PostBookmark, \
-    PostReplyBookmark, CommunityBlock
+    PostReplyBookmark, CommunityBlock, File
 from app.post import bp
 from app.utils import get_setting, render_template, allowlist_html, markdown_to_html, validation_required, \
     shorten_string, markdown_to_text, gibberish, ap_datetime, return_304, \
@@ -32,7 +32,7 @@ from app.utils import get_setting, render_template, allowlist_html, markdown_to_
     blocked_instances, blocked_domains, community_moderators, blocked_phrases, show_ban_message, recently_upvoted_posts, \
     recently_downvoted_posts, recently_upvoted_post_replies, recently_downvoted_post_replies, reply_is_stupid, \
     languages_for_form, menu_topics, add_to_modlog, blocked_communities, piefed_markdown_to_lemmy_markdown, \
-    permission_required, blocked_users
+    permission_required, blocked_users, get_request
 
 
 def show_post(post_id: int):
@@ -109,6 +109,9 @@ def show_post(post_id: int):
             'language': {
                 'identifier': reply.language_code(),
                 'name': reply.language_name()
+            },
+            'contentMap': {
+                reply.language_code(): reply.body_html
             }
         }
         create_json = {
@@ -130,8 +133,8 @@ def show_post(post_id: int):
             }]
         }
         if not community.is_local():    # this is a remote community, send it to the instance that hosts it
-            success = post_request(community.ap_inbox_url, create_json, current_user.private_key,
+            success = post_request_in_background(community.ap_inbox_url, create_json, current_user.private_key,
-                                                       current_user.public_url() + '#main-key')
+                                                 current_user.public_url() + '#main-key', timeout=10)
             if success is False or isinstance(success, str):
                 flash('Failed to send to remote instance', 'error')
         else:                       # local community - send it to followers on remote instances
@@ -156,13 +159,13 @@ def show_post(post_id: int):
         # send copy of Note to post author (who won't otherwise get it if no-one else on their instance is subscribed to the community)
         if not post.author.is_local() and post.author.ap_domain != community.ap_domain:
             if not community.is_local() or (community.is_local and not community.has_followers_from_domain(post.author.ap_domain)):
-                success = post_request(post.author.ap_inbox_url, create_json, current_user.private_key,
+                success = post_request_in_background(post.author.ap_inbox_url, create_json, current_user.private_key,
-                                                       current_user.public_url() + '#main-key')
+                                                     current_user.public_url() + '#main-key', timeout=10)
                 if success is False or isinstance(success, str):
                     # sending to shared inbox is good enough for Mastodon, but Lemmy will reject it the local community has no followers
                     personal_inbox = post.author.public_url() + '/inbox'
-                    post_request(personal_inbox, create_json, current_user.private_key,
+                    post_request_in_background(personal_inbox, create_json, current_user.private_key,
-                                                       current_user.public_url() + '#main-key')
+                                               current_user.public_url() + '#main-key', timeout=10)
 
         return redirect(url_for('activitypub.post_ap', post_id=post_id, _anchor=f'comment_{reply.id}'))
     else:
@@ -272,6 +275,7 @@ def show_post(post_id: int):
                            inoculation=inoculation[randint(0, len(inoculation) - 1)] if g.site.show_inoculation_block else None
                            )
     response.headers.set('Vary', 'Accept, Cookie, Accept-Language')
+    response.headers.set('Link', f'<https://{current_app.config["SERVER_NAME"]}/post/{post.id}>; rel="alternate"; type="application/activity+json"')
     return response
 
 
@@ -466,8 +470,13 @@ def poll_vote(post_id):
 def continue_discussion(post_id, comment_id):
     post = Post.query.get_or_404(post_id)
     comment = PostReply.query.get_or_404(comment_id)
+
     if post.community.banned or post.deleted or comment.deleted:
-        abort(404)
+        if current_user.is_anonymous or not (current_user.is_authenticated and (current_user.is_admin() or current_user.is_staff())):
+            abort(404)
+        else:
+            flash(_('This comment has been deleted and is only visible to staff and admins.'), 'warning')
+
     mods = post.community.moderators()
     is_moderator = current_user.is_authenticated and any(mod.user_id == current_user.id for mod in mods)
     if post.community.private_mods:
@@ -492,7 +501,7 @@ def continue_discussion(post_id, comment_id):
 @bp.route('/post/<int:post_id>/comment/<int:comment_id>/reply', methods=['GET', 'POST'])
 @login_required
 def add_reply(post_id: int, comment_id: int):
-    if current_user.banned:
+    if current_user.banned or current_user.ban_comments:
         return show_ban_message()
     post = Post.query.get_or_404(post_id)
 
@@ -583,6 +592,10 @@ def add_reply(post_id: int, comment_id: int):
                 'published': ap_datetime(utcnow()),
                 'distinguished': False,
                 'audience': post.community.public_url(),
+                'language': {
+                    'identifier': reply.language_code(),
+                    'name': reply.language_name()
+                },
                 'contentMap': {
                     'en': reply.body_html
                 }
@@ -1888,6 +1901,35 @@ def post_reply_view_voting_activity(comment_id: int):
                            )
 
 
+@bp.route('/post/<int:post_id>/fixup_from_remote', methods=['GET'])
+@login_required
+@permission_required('change instance settings')
+def post_fixup_from_remote(post_id: int):
+    post = Post.query.get_or_404(post_id)
+
+    # will fail for some MBIN objects for same reason that 'View original on ...' does
+    # (ap_id is lowercase, but original URL was mixed-case and remote instance software is case-sensitive)
+    remote_post_request = get_request(post.ap_id, headers={'Accept': 'application/activity+json'})
+    if remote_post_request.status_code == 200:
+        remote_post_json = remote_post_request.json()
+        remote_post_request.close()
+        if 'type' in remote_post_json and remote_post_json['type'] == 'Page':
+            post.domain_id = None
+            file_entry_to_delete = None
+            if post.image_id:
+                file_entry_to_delete = post.image_id
+            post.image_id = None
+            post.url = None
+            db.session.commit()
+            if file_entry_to_delete:
+                File.query.filter_by(id=file_entry_to_delete).delete()
+                db.session.commit()
+            update_json = {'type': 'Update', 'object': remote_post_json}
+            update_post_from_activity(post, update_json)
+
+    return redirect(url_for('activitypub.post_ap', post_id=post.id))
+
+
 @bp.route('/post/<int:post_id>/cross-post', methods=['GET', 'POST'])
 @login_required
 def post_cross_post(post_id: int):

app/shared/community.py

@@ -1,171 +1,66 @@
 from app import db, cache
-from app.activitypub.signature import post_request
 from app.constants import *
-from app.models import Community, CommunityBan, CommunityBlock, CommunityJoinRequest, CommunityMember
+from app.models import CommunityBlock, CommunityMember
-from app.utils import authorise_api_user, blocked_communities, community_membership, joined_communities, gibberish
+from app.shared.tasks import task_selector
+from app.utils import authorise_api_user, blocked_communities
 
-from flask import abort, current_app, flash
+from flask import current_app, flash
 from flask_babel import _
 from flask_login import current_user
 
+
 # would be in app/constants.py
 SRC_WEB = 1
 SRC_PUB = 2
 SRC_API = 3
+SRC_PLD = 4     # admin preload form to seed communities
+
 
 # function can be shared between WEB and API (only API calls it for now)
 # call from admin.federation not tested
-def join_community(community_id: int, src, auth=None, user_id=None, main_user_name=True):
+def join_community(community_id: int, src, auth=None, user_id=None):
     if src == SRC_API:
-        community = Community.query.filter_by(id=community_id).one()
+        user_id = authorise_api_user(auth)
-        user = authorise_api_user(auth, return_type='model')
-    else:
-        community = Community.query.get_or_404(community_id)
-        if not user_id:
-            user = current_user
-        else:
-            user = User.query.get(user_id)
 
-    pre_load_message = {}
+    send_async = not (current_app.debug or src == SRC_WEB)     # False if using a browser
-    if community_membership(user, community) != SUBSCRIPTION_MEMBER and community_membership(user, community) != SUBSCRIPTION_PENDING:
-        banned = CommunityBan.query.filter_by(user_id=user.id, community_id=community.id).first()
-        if banned:
-            if src == SRC_API:
-                raise Exception('banned_from_community')
-            else:
-                if main_user_name:
-                    flash(_('You cannot join this community'))
-                    return
-                else:
-                    pre_load_message['user_banned'] = True
-                    return pre_load_message
-    else:
-        if src == SRC_API:
-            return user.id
-        else:
-            if not main_user_name:
-                pre_load_message['status'] = 'already subscribed, or subsciption pending'
-                return pre_load_message
 
-    success = True
+    sync_retval = task_selector('join_community', send_async, user_id=user_id, community_id=community_id, src=src)
-    remote = not community.is_local()
+
-    if remote:
+    if send_async or sync_retval is True:
-        # send ActivityPub message to remote community, asking to follow. Accept message will be sent to our shared inbox
+        member = CommunityMember(user_id=user_id, community_id=community_id)
-        join_request = CommunityJoinRequest(user_id=user.id, community_id=community.id)
+        db.session.add(member)
-        db.session.add(join_request)
         db.session.commit()
-        if community.instance.online():
-            follow = {
-              "actor": user.public_url(main_user_name=main_user_name),
-              "to": [community.public_url()],
-              "object": community.public_url(),
-              "type": "Follow",
-              "id": f"https://{current_app.config['SERVER_NAME']}/activities/follow/{join_request.id}"
-            }
-            success = post_request(community.ap_inbox_url, follow, user.private_key,
-                                    user.public_url(main_user_name=main_user_name) + '#main-key', timeout=10)
-            if success is False or isinstance(success, str):
-                if 'is not in allowlist' in success:
-                    if src == SRC_API:
-                        raise Exception('not_in_remote_instance_allowlist')
-                    else:
-                        msg_to_user = f'{community.instance.domain} does not allow us to join their communities.'
-                        if main_user_name:
-                            flash(_(msg_to_user), 'error')
-                            return
-                        else:
-                            pre_load_message['status'] = msg_to_user
-                            return pre_load_message
-                else:
-                    if src != SRC_API:
-                        msg_to_user = "There was a problem while trying to communicate with remote server. If other people have already joined this community it won't matter."
-                        if main_user_name:
-                            flash(_(msg_to_user), 'error')
-                            return
-                        else:
-                            pre_load_message['status'] = msg_to_user
-                            return pre_load_message
 
-    # for local communities, joining is instant
+    if src == SRC_API:
-    member = CommunityMember(user_id=user.id, community_id=community.id)
+        return user_id
-    db.session.add(member)
+    elif src == SRC_PLD:
-    db.session.commit()
+        return sync_retval
-    if success is True:
+    else:
-        cache.delete_memoized(community_membership, user, community)
+        return
-        cache.delete_memoized(joined_communities, user.id)
-        if src == SRC_API:
-            return user.id
-        else:
-            if main_user_name:
-                flash('You joined ' + community.title)
-            else:
-                pre_load_message['status'] = 'joined'
-
-            if not main_user_name:
-                return pre_load_message
-
-    # for SRC_WEB, calling function should handle if the community isn't found
 
 
 # function can be shared between WEB and API (only API calls it for now)
 def leave_community(community_id: int, src, auth=None):
-    if src == SRC_API:
+    user_id = authorise_api_user(auth) if src == SRC_API else current_user.id
-        community = Community.query.filter_by(id=community_id).one()
+    cm = CommunityMember.query.filter_by(user_id=user_id, community_id=community_id).one()
-        user = authorise_api_user(auth, return_type='model')
+    if not cm.is_owner:
+        task_selector('leave_community', user_id=user_id, community_id=community_id)
+
+        db.session.query(CommunityMember).filter_by(user_id=user_id, community_id=community_id).delete()
+        db.session.commit()
+
+        if src == SRC_WEB:
+            flash('You have left the community')
     else:
-        community = Community.query.get_or_404(community_id)
+        # todo: community deletion
-        user = current_user
+        if src == SRC_API:
-
+            raise Exception('need_to_make_someone_else_owner')
-    subscription = community_membership(user, community)
-    if subscription:
-        if subscription != SUBSCRIPTION_OWNER:
-            proceed = True
-            # Undo the Follow
-            if not community.is_local():
-                success = True
-                if not community.instance.gone_forever:
-                    undo_id = f"https://{current_app.config['SERVER_NAME']}/activities/undo/" + gibberish(15)
-                    follow = {
-                      "actor": user.public_url(),
-                      "to": [community.public_url()],
-                      "object": community.public_url(),
-                      "type": "Follow",
-                      "id": f"https://{current_app.config['SERVER_NAME']}/activities/follow/{gibberish(15)}"
-                    }
-                    undo = {
-                      'actor': user.public_url(),
-                      'to': [community.public_url()],
-                      'type': 'Undo',
-                      'id': undo_id,
-                      'object': follow
-                    }
-                    success = post_request(community.ap_inbox_url, undo, user.private_key,
-                                              user.public_url() + '#main-key', timeout=10)
-                    if success is False or isinstance(success, str):
-                        if src != SRC_API:
-                            flash('There was a problem while trying to unsubscribe', 'error')
-                            return
-
-            if proceed:
-                db.session.query(CommunityMember).filter_by(user_id=user.id, community_id=community.id).delete()
-                db.session.query(CommunityJoinRequest).filter_by(user_id=user.id, community_id=community.id).delete()
-                db.session.commit()
-
-                if src != SRC_API:
-                    flash('You have left ' + community.title)
-
-            cache.delete_memoized(community_membership, user, community)
-            cache.delete_memoized(joined_communities, user.id)
         else:
-            # todo: community deletion
+            flash('You need to make someone else the owner before unsubscribing.', 'warning')
-            if src == SRC_API:
+            return
-                raise Exception('need_to_make_someone_else_owner')
-            else:
-                flash('You need to make someone else the owner before unsubscribing.', 'warning')
-                return
 
     if src == SRC_API:
-        return user.id
+        return user_id
     else:
         # let calling function handle redirect
         return

app/shared/post.py

@@ -1,11 +1,10 @@
-from app import cache, db
+from app import db
-from app.activitypub.signature import default_context, post_request_in_background
-from app.community.util import send_to_remote_instance
 from app.constants import *
-from app.models import NotificationSubscription, Post, PostBookmark, User
+from app.models import NotificationSubscription, Post, PostBookmark
-from app.utils import gibberish, instance_banned, render_template, authorise_api_user, recently_upvoted_posts, recently_downvoted_posts, shorten_string
+from app.shared.tasks import task_selector
+from app.utils import render_template, authorise_api_user, shorten_string
 
-from flask import abort, current_app, flash, redirect, request, url_for
+from flask import abort, flash, redirect, request, url_for
 from flask_babel import _
 from flask_login import current_user
 
@@ -28,51 +27,7 @@ def vote_for_post(post_id: int, vote_direction, src, auth=None):
 
     undo = post.vote(user, vote_direction)
 
-    if not post.community.local_only:
+    task_selector('vote_for_post', user_id=user.id, post_id=post_id, vote_to_undo=undo, vote_direction=vote_direction)
-        if undo:
-            action_json = {
-                'actor': user.public_url(not(post.community.instance.votes_are_public() and user.vote_privately())),
-                'type': 'Undo',
-                'id': f"https://{current_app.config['SERVER_NAME']}/activities/undo/{gibberish(15)}",
-                'audience': post.community.public_url(),
-                'object': {
-                    'actor': user.public_url(not(post.community.instance.votes_are_public() and user.vote_privately())),
-                    'object': post.public_url(),
-                    'type': undo,
-                    'id': f"https://{current_app.config['SERVER_NAME']}/activities/{undo.lower()}/{gibberish(15)}",
-                    'audience': post.community.public_url()
-                }
-            }
-        else:
-            action_type = 'Like' if vote_direction == 'upvote' else 'Dislike'
-            action_json = {
-                'actor': user.public_url(not(post.community.instance.votes_are_public() and user.vote_privately())),
-                'object': post.profile_id(),
-                'type': action_type,
-                'id': f"https://{current_app.config['SERVER_NAME']}/activities/{action_type.lower()}/{gibberish(15)}",
-                'audience': post.community.public_url()
-            }
-        if post.community.is_local():
-            announce = {
-                    "id": f"https://{current_app.config['SERVER_NAME']}/activities/announce/{gibberish(15)}",
-                    "type": 'Announce',
-                    "to": [
-                        "https://www.w3.org/ns/activitystreams#Public"
-                    ],
-                    "actor": post.community.public_url(),
-                    "cc": [
-                        post.community.ap_followers_url
-                    ],
-                    '@context': default_context(),
-                    'object': action_json
-            }
-            for instance in post.community.following_instances():
-                if instance.inbox and not user.has_blocked_instance(instance.id) and not instance_banned(instance.domain):
-                    send_to_remote_instance(instance.id, post.community.id, announce)
-        else:
-            post_request_in_background(post.community.ap_inbox_url, action_json, user.private_key,
-                                       user.public_url(not(post.community.instance.votes_are_public() and user.vote_privately())) + '#main-key')
-
 
     if src == SRC_API:
         return user.id
@@ -83,8 +38,6 @@ def vote_for_post(post_id: int, vote_direction, src, auth=None):
             recently_upvoted = [post_id]
         elif vote_direction == 'downvote' and undo is None:
             recently_downvoted = [post_id]
-        cache.delete_memoized(recently_upvoted_posts, user.id)
-        cache.delete_memoized(recently_downvoted_posts, user.id)
 
         template = 'post/_post_voting_buttons.html' if request.args.get('style', '') == '' else 'post/_post_voting_buttons_masonry.html'
         return render_template(template, post=post, community=post.community, recently_upvoted=recently_upvoted,

app/shared/reply.py

@@ -3,6 +3,7 @@ from app.activitypub.signature import default_context, post_request_in_backgroun
 from app.community.util import send_to_remote_instance
 from app.constants import *
 from app.models import Instance, Notification, NotificationSubscription, Post, PostReply, PostReplyBookmark, Report, Site, User, utcnow
+from app.shared.tasks import task_selector
 from app.utils import gibberish, instance_banned, render_template, authorise_api_user, recently_upvoted_post_replies, recently_downvoted_post_replies, shorten_string, \
                       piefed_markdown_to_lemmy_markdown, markdown_to_html, ap_datetime
 
@@ -18,7 +19,6 @@ SRC_API = 3
 
 # function can be shared between WEB and API (only API calls it for now)
 # comment_vote in app/post/routes would just need to do 'return vote_for_reply(reply_id, vote_direction, SRC_WEB)'
-
 def vote_for_reply(reply_id: int, vote_direction, src, auth=None):
     if src == SRC_API:
         reply = PostReply.query.filter_by(id=reply_id).one()
@@ -29,50 +29,7 @@ def vote_for_reply(reply_id: int, vote_direction, src, auth=None):
 
     undo = reply.vote(user, vote_direction)
 
-    if not reply.community.local_only:
+    task_selector('vote_for_reply', user_id=user.id, reply_id=reply_id, vote_to_undo=undo, vote_direction=vote_direction)
-        if undo:
-            action_json = {
-                'actor': user.public_url(not(reply.community.instance.votes_are_public() and user.vote_privately())),
-                'type': 'Undo',
-                'id': f"https://{current_app.config['SERVER_NAME']}/activities/undo/{gibberish(15)}",
-                'audience': reply.community.public_url(),
-                'object': {
-                    'actor': user.public_url(not(reply.community.instance.votes_are_public() and user.vote_privately())),
-                    'object': reply.public_url(),
-                    'type': undo,
-                    'id': f"https://{current_app.config['SERVER_NAME']}/activities/{undo.lower()}/{gibberish(15)}",
-                    'audience': reply.community.public_url()
-                }
-            }
-        else:
-            action_type = 'Like' if vote_direction == 'upvote' else 'Dislike'
-            action_json = {
-                'actor': user.public_url(not(reply.community.instance.votes_are_public() and user.vote_privately())),
-                'object': reply.public_url(),
-                'type': action_type,
-                'id': f"https://{current_app.config['SERVER_NAME']}/activities/{action_type.lower()}/{gibberish(15)}",
-                'audience': reply.community.public_url()
-            }
-        if reply.community.is_local():
-            announce = {
-                    "id": f"https://{current_app.config['SERVER_NAME']}/activities/announce/{gibberish(15)}",
-                    "type": 'Announce',
-                    "to": [
-                        "https://www.w3.org/ns/activitystreams#Public"
-                    ],
-                    "actor": reply.community.ap_profile_id,
-                    "cc": [
-                        reply.community.ap_followers_url
-                    ],
-                    '@context': default_context(),
-                    'object': action_json
-            }
-            for instance in reply.community.following_instances():
-                if instance.inbox and not user.has_blocked_instance(instance.id) and not instance_banned(instance.domain):
-                    send_to_remote_instance(instance.id, reply.community.id, announce)
-        else:
-            post_request_in_background(reply.community.ap_inbox_url, action_json, user.private_key,
-                                       user.public_url(not(reply.community.instance.votes_are_public() and user.vote_privately())) + '#main-key')
 
     if src == SRC_API:
         return user.id
@@ -83,8 +40,6 @@ def vote_for_reply(reply_id: int, vote_direction, src, auth=None):
             recently_upvoted = [reply_id]
         elif vote_direction == 'downvote' and undo is None:
             recently_downvoted = [reply_id]
-        cache.delete_memoized(recently_upvoted_post_replies, user.id)
-        cache.delete_memoized(recently_downvoted_post_replies, user.id)
 
         return render_template('post/_reply_voting_buttons.html', comment=reply,
                                recently_upvoted_replies=recently_upvoted,
@@ -206,8 +161,8 @@ def basic_rate_limit_check(user):
 def make_reply(input, post, parent_id, src, auth=None):
     if src == SRC_API:
         user = authorise_api_user(auth, return_type='model')
-        if not basic_rate_limit_check(user):
+        #if not basic_rate_limit_check(user):
-            raise Exception('rate_limited')
+        #    raise Exception('rate_limited')
         content = input['body']
         notify_author = input['notify_author']
         language_id = input['language_id']
@@ -235,104 +190,7 @@ def make_reply(input, post, parent_id, src, auth=None):
         input.body.data = ''
         flash('Your comment has been added.')
 
-    # federation
+    task_selector('make_reply', user_id=user.id, reply_id=reply.id, parent_id=parent_id)
-    if parent_id:
-        in_reply_to = parent_reply
-    else:
-        in_reply_to = post
-
-    if not post.community.local_only:
-        reply_json = {
-          'type': 'Note',
-          'id': reply.public_url(),
-          'attributedTo': user.public_url(),
-          'to': [
-            'https://www.w3.org/ns/activitystreams#Public'
-          ],
-          'cc': [
-            post.community.public_url(),
-            in_reply_to.author.public_url()
-          ],
-          'content': reply.body_html,
-          'inReplyTo': in_reply_to.profile_id(),
-          'url': reply.profile_id(),
-          'mediaType': 'text/html',
-          'source': {'content': reply.body, 'mediaType': 'text/markdown'},
-          'published': ap_datetime(utcnow()),
-          'distinguished': False,
-          'audience': post.community.public_url(),
-          'contentMap': {
-            'en': reply.body_html
-          },
-          'language': {
-            'identifier': reply.language_code(),
-            'name': reply.language_name()
-          }
-        }
-        create_json = {
-          '@context': default_context(),
-          'type': 'Create',
-          'actor': user.public_url(),
-          'audience': post.community.public_url(),
-          'to': [
-            'https://www.w3.org/ns/activitystreams#Public'
-          ],
-          'cc': [
-            post.community.public_url(),
-            in_reply_to.author.public_url()
-          ],
-          'object': reply_json,
-          'id': f"https://{current_app.config['SERVER_NAME']}/activities/create/{gibberish(15)}"
-        }
-        if in_reply_to.notify_author and in_reply_to.author.ap_id is not None:
-            reply_json['tag'] = [
-              {
-                'href': in_reply_to.author.public_url(),
-                'name': in_reply_to.author.mention_tag(),
-                'type': 'Mention'
-              }
-            ]
-            create_json['tag'] = [
-              {
-                'href': in_reply_to.author.public_url(),
-                'name': in_reply_to.author.mention_tag(),
-                'type': 'Mention'
-              }
-            ]
-        if not post.community.is_local():    # this is a remote community, send it to the instance that hosts it
-            success = post_request(post.community.ap_inbox_url, create_json, user.private_key,
-                                    user.public_url() + '#main-key')
-            if src == SRC_WEB:
-                if success is False or isinstance(success, str):
-                    flash('Failed to send reply', 'error')
-        else:                                # local community - send it to followers on remote instances
-            del create_json['@context']
-            announce = {
-              'id': f"https://{current_app.config['SERVER_NAME']}/activities/announce/{gibberish(15)}",
-              'type': 'Announce',
-              'to': [
-                'https://www.w3.org/ns/activitystreams#Public'
-              ],
-              'actor': post.community.public_url(),
-              'cc': [
-                post.community.ap_followers_url
-              ],
-              '@context': default_context(),
-              'object': create_json
-            }
-            for instance in post.community.following_instances():
-                if instance.inbox and not user.has_blocked_instance(instance.id) and not instance_banned(instance.domain):
-                    send_to_remote_instance(instance.id, post.community.id, announce)
-
-        # send copy of Note to comment author (who won't otherwise get it if no-one else on their instance is subscribed to the community)
-        if not in_reply_to.author.is_local() and in_reply_to.author.ap_domain != reply.community.ap_domain:
-            if not post.community.is_local() or (post.community.is_local and not post.community.has_followers_from_domain(in_reply_to.author.ap_domain)):
-                success = post_request(in_reply_to.author.ap_inbox_url, create_json, user.private_key, user.public_url() + '#main-key')
-                if success is False or isinstance(success, str):
-                    # sending to shared inbox is good enough for Mastodon, but Lemmy will reject it the local community has no followers
-                    personal_inbox = in_reply_to.author.public_url() + '/inbox'
-                    post_request(personal_inbox, create_json, user.private_key, user.public_url() + '#main-key')
-
 
     if src == SRC_API:
         return user.id, reply
@@ -364,105 +222,7 @@ def edit_reply(input, reply, post, src, auth=None):
     if src == SRC_WEB:
         flash(_('Your changes have been saved.'), 'success')
 
-    if reply.parent_id:
+    task_selector('edit_reply', user_id=user.id, reply_id=reply.id, parent_id=reply.parent_id)
-        in_reply_to = PostReply.query.filter_by(id=reply.parent_id).one()
-    else:
-        in_reply_to = post
-
-    # federate edit
-    if not post.community.local_only:
-        reply_json = {
-          'type': 'Note',
-          'id': reply.public_url(),
-          'attributedTo': user.public_url(),
-          'to': [
-            'https://www.w3.org/ns/activitystreams#Public'
-          ],
-          'cc': [
-            post.community.public_url(),
-            in_reply_to.author.public_url()
-          ],
-          'content': reply.body_html,
-          'inReplyTo': in_reply_to.profile_id(),
-          'url': reply.public_url(),
-          'mediaType': 'text/html',
-          'source': {'content': reply.body, 'mediaType': 'text/markdown'},
-          'published': ap_datetime(reply.posted_at),
-          'updated': ap_datetime(reply.edited_at),
-          'distinguished': False,
-          'audience': post.community.public_url(),
-          'contentMap': {
-            'en': reply.body_html
-          },
-          'language': {
-            'identifier': reply.language_code(),
-            'name': reply.language_name()
-          }
-        }
-        update_json = {
-          '@context': default_context(),
-          'type': 'Update',
-          'actor': user.public_url(),
-          'audience': post.community.public_url(),
-          'to': [
-            'https://www.w3.org/ns/activitystreams#Public'
-          ],
-          'cc': [
-            post.community.public_url(),
-            in_reply_to.author.public_url()
-          ],
-          'object': reply_json,
-          'id': f"https://{current_app.config['SERVER_NAME']}/activities/update/{gibberish(15)}"
-        }
-        if in_reply_to.notify_author and in_reply_to.author.ap_id is not None:
-            reply_json['tag'] = [
-              {
-                'href': in_reply_to.author.public_url(),
-                'name': in_reply_to.author.mention_tag(),
-                'type': 'Mention'
-              }
-            ]
-            update_json['tag'] = [
-              {
-                'href': in_reply_to.author.public_url(),
-                'name': in_reply_to.author.mention_tag(),
-                'type': 'Mention'
-              }
-            ]
-        if not post.community.is_local():    # this is a remote community, send it to the instance that hosts it
-            success = post_request(post.community.ap_inbox_url, update_json, user.private_key,
-                                                               user.public_url() + '#main-key')
-            if src == SRC_WEB:
-                if success is False or isinstance(success, str):
-                    flash('Failed to send send edit to remote server', 'error')
-        else:                                # local community - send it to followers on remote instances
-            del update_json['@context']
-            announce = {
-              'id': f"https://{current_app.config['SERVER_NAME']}/activities/announce/{gibberish(15)}",
-              'type': 'Announce',
-              'to': [
-                'https://www.w3.org/ns/activitystreams#Public'
-              ],
-              'actor': post.community.public_url(),
-              'cc': [
-                post.community.ap_followers_url
-              ],
-              '@context': default_context(),
-              'object': update_json
-            }
-
-            for instance in post.community.following_instances():
-                if instance.inbox and not user.has_blocked_instance(instance.id) and not instance_banned(instance.domain):
-                    send_to_remote_instance(instance.id, post.community.id, announce)
-
-        # send copy of Note to post author (who won't otherwise get it if no-one else on their instance is subscribed to the community)
-        if not in_reply_to.author.is_local() and in_reply_to.author.ap_domain != reply.community.ap_domain:
-            if not post.community.is_local() or (post.community.is_local and not post.community.has_followers_from_domain(in_reply_to.author.ap_domain)):
-                success = post_request(in_reply_to.author.ap_inbox_url, update_json, user.private_key, user.public_url() + '#main-key')
-                if success is False or isinstance(success, str):
-                    # sending to shared inbox is good enough for Mastodon, but Lemmy will reject it the local community has no followers
-                    personal_inbox = in_reply_to.author.public_url() + '/inbox'
-                    post_request(personal_inbox, update_json, user.private_key, user.public_url() + '#main-key')
 
     if src == SRC_API:
         return user.id, reply
@@ -471,77 +231,28 @@ def edit_reply(input, reply, post, src, auth=None):
 
 
 # just for deletes by owner (mod deletes are classed as 'remove')
-# just for API for now, as WEB version needs attention to ensure that replies can be 'undeleted'
 def delete_reply(reply_id, src, auth):
     if src == SRC_API:
         reply = PostReply.query.filter_by(id=reply_id, deleted=False).one()
-        post = Post.query.filter_by(id=reply.post_id).one()
+        user_id = authorise_api_user(auth, id_match=reply.user_id)
-        user = authorise_api_user(auth, return_type='model', id_match=reply.user_id)
     else:
         reply = PostReply.query.get_or_404(reply_id)
-        post = Post.query.get_or_404(reply.post_id)
+        user_id = current_user.id
-        user = current_user
 
     reply.deleted = True
-    reply.deleted_by = user.id
+    reply.deleted_by = user_id
-    # everything else (votes, body, reports, bookmarks, subscriptions, etc) only wants deleting when it's properly purged after 7 days
-    # reply_view will return '' in body if reply.deleted == True
 
     if not reply.author.bot:
-        post.reply_count -= 1
+        reply.post.reply_count -= 1
     reply.author.post_reply_count -= 1
     db.session.commit()
     if src == SRC_WEB:
         flash(_('Comment deleted.'))
 
-    # federate delete
+    task_selector('delete_reply', user_id=user_id, reply_id=reply.id)
-    if not post.community.local_only:
-        delete_json = {
-          'id': f"https://{current_app.config['SERVER_NAME']}/activities/delete/{gibberish(15)}",
-          'type': 'Delete',
-          'actor': user.public_url(),
-          'audience': post.community.public_url(),
-          'to': [
-            'https://www.w3.org/ns/activitystreams#Public'
-          ],
-          'published': ap_datetime(utcnow()),
-          'cc': [
-            post.community.public_url(),
-            user.followers_url()
-          ],
-          'object': reply.ap_id,
-          '@context': default_context()
-        }
-
-        if not post.community.is_local():   # this is a remote community, send it to the instance that hosts it
-            success = post_request(post.community.ap_inbox_url, delete_json, user.private_key,
-                                        user.public_url() + '#main-key')
-            if src == SRC_WEB:
-                if success is False or isinstance(success, str):
-                    flash('Failed to send delete to remote server', 'error')
-
-        else:                               # local community - send it to followers on remote instances
-            del delete_json['@context']
-            announce = {
-              'id': f"https://{current_app.config['SERVER_NAME']}/activities/announce/{gibberish(15)}",
-              'type': 'Announce',
-              'to': [
-                'https://www.w3.org/ns/activitystreams#Public'
-              ],
-              'actor': post.community.public_url(),
-              'cc': [
-                post.community.public_url() + '/followers'
-              ],
-              '@context': default_context(),
-              'object': delete_json
-            }
-
-            for instance in post.community.following_instances():
-                if instance.inbox:
-                    send_to_remote_instance(instance.id, post.community.id, announce)
 
     if src == SRC_API:
-        return user.id, reply
+        return user_id, reply
     else:
         return
 
@@ -549,87 +260,27 @@ def delete_reply(reply_id, src, auth):
 def restore_reply(reply_id, src, auth):
     if src == SRC_API:
         reply = PostReply.query.filter_by(id=reply_id, deleted=True).one()
-        post = Post.query.filter_by(id=reply.post_id).one()
+        user_id = authorise_api_user(auth, id_match=reply.user_id)
-        user = authorise_api_user(auth, return_type='model', id_match=reply.user_id)
+        if reply.user_id != reply.deleted_by:
-        if reply.deleted_by and reply.user_id != reply.deleted_by:
             raise Exception('incorrect_login')
     else:
         reply = PostReply.query.get_or_404(reply_id)
-        post = Post.query.get_or_404(reply.post_id)
+        user_id = current_user.id
-        user = current_user
 
     reply.deleted = False
     reply.deleted_by = None
 
     if not reply.author.bot:
-        post.reply_count += 1
+        reply.post.reply_count += 1
     reply.author.post_reply_count += 1
     db.session.commit()
     if src == SRC_WEB:
         flash(_('Comment restored.'))
 
-    # federate undelete
+    task_selector('restore_reply', user_id=user_id, reply_id=reply.id)
-    if not post.community.local_only:
-        delete_json = {
-          'id': f"https://{current_app.config['SERVER_NAME']}/activities/delete/{gibberish(15)}",
-          'type': 'Delete',
-          'actor': user.public_url(),
-          'audience': post.community.public_url(),
-          'to': [
-            'https://www.w3.org/ns/activitystreams#Public'
-          ],
-          'published': ap_datetime(utcnow()),
-          'cc': [
-            post.community.public_url(),
-            user.followers_url()
-          ],
-          'object': reply.ap_id
-        }
-        undo_json = {
-          'id': f"https://{current_app.config['SERVER_NAME']}/activities/undo/{gibberish(15)}",
-          'type': 'Undo',
-          'actor': user.public_url(),
-          'audience': post.community.public_url(),
-          'to': [
-            'https://www.w3.org/ns/activitystreams#Public'
-          ],
-          'cc': [
-            post.community.public_url(),
-            user.followers_url()
-          ],
-          'object': delete_json,
-          '@context': default_context()
-        }
-
-        if not post.community.is_local():   # this is a remote community, send it to the instance that hosts it
-            success = post_request(post.community.ap_inbox_url, undo_json, user.private_key,
-                                        user.public_url() + '#main-key')
-            if src == SRC_WEB:
-                if success is False or isinstance(success, str):
-                    flash('Failed to send delete to remote server', 'error')
-
-        else:                               # local community - send it to followers on remote instances
-            del undo_json['@context']
-            announce = {
-              'id': f"https://{current_app.config['SERVER_NAME']}/activities/announce/{gibberish(15)}",
-              'type': 'Announce',
-              'to': [
-                'https://www.w3.org/ns/activitystreams#Public'
-              ],
-              'actor': post.community.public_url(),
-              'cc': [
-                post.community.public_url() + '/followers'
-              ],
-              '@context': default_context(),
-              'object': undo_json
-            }
-
-            for instance in post.community.following_instances():
-                if instance.inbox:
-                    send_to_remote_instance(instance.id, post.community.id, announce)
 
     if src == SRC_API:
-        return user.id, reply
+        return user_id, reply
     else:
         return
 
@@ -637,13 +288,13 @@ def restore_reply(reply_id, src, auth):
 def report_reply(reply_id, input, src, auth=None):
     if src == SRC_API:
         reply = PostReply.query.filter_by(id=reply_id).one()
-        user = authorise_api_user(auth, return_type='model')
+        user_id = authorise_api_user(auth)
         reason = input['reason']
         description = input['description']
         report_remote = input['report_remote']
     else:
         reply = PostReply.query.get_or_404(reply_id)
-        user = current_user
+        user_id = current_user.id
         reason = input.reasons_to_string(input.reasons.data)
         description = input.description.data
         report_remote = input.report_remote.data
@@ -655,7 +306,7 @@ def report_reply(reply_id, input, src, auth=None):
             flash(_('Comment has already been reported, thank you!'))
             return
 
-    report = Report(reasons=reason, description=description, type=2, reporter_id=user.id, suspect_post_id=reply.post.id, suspect_community_id=reply.community.id,
+    report = Report(reasons=reason, description=description, type=2, reporter_id=user_id, suspect_post_id=reply.post.id, suspect_community_id=reply.community.id,
                     suspect_user_id=reply.author.id, suspect_post_reply_id=reply.id, in_community_id=reply.community.id, source_instance_id=1)
     db.session.add(report)
 
@@ -666,14 +317,14 @@ def report_reply(reply_id, input, src, auth=None):
         if moderator and moderator.is_local():
             notification = Notification(user_id=mod.user_id, title=_('A comment has been reported'),
                                         url=f"https://{current_app.config['SERVER_NAME']}/comment/{reply.id}",
-                                        author_id=user.id)
+                                        author_id=user_id)
             db.session.add(notification)
             already_notified.add(mod.user_id)
     reply.reports += 1
     # todo: only notify admins for certain types of report
     for admin in Site.admins():
         if admin.id not in already_notified:
-            notify = Notification(title='Suspicious content', url='/admin/reports', user_id=admin.id, author_id=user.id)
+            notify = Notification(title='Suspicious content', url='/admin/reports', user_id=admin.id, author_id=user_id)
             db.session.add(notify)
             admin.unread_notifications += 1
     db.session.commit()
@@ -683,26 +334,10 @@ def report_reply(reply_id, input, src, auth=None):
         summary = reason
         if description:
             summary += ' - ' + description
-        report_json = {
+
-          'actor': user.public_url(),
+        task_selector('report_reply', user_id=user_id, reply_id=reply_id, summary=summary)
-          'audience': reply.community.public_url(),
-          'content': None,
-          'id': f"https://{current_app.config['SERVER_NAME']}/activities/flag/{gibberish(15)}",
-          'object': reply.ap_id,
-          'summary': summary,
-          'to': [
-            reply.community.public_url()
-          ],
-          'type': 'Flag'
-        }
-        instance = Instance.query.get(reply.community.instance_id)
-        if reply.community.ap_inbox_url and not user.has_blocked_instance(instance.id) and not instance_banned(instance.domain):
-            success = post_request(reply.community.ap_inbox_url, report_json, user.private_key, user.public_url() + '#main-key')
-            if success is False or isinstance(success, str):
-                if src == SRC_WEB:
-                    flash('Failed to send report to remote server', 'error')
 
     if src == SRC_API:
-        return user.id, report
+        return user_id, report
     else:
         return

app/shared/tasks/__init__.py

@@ -0,0 +1,30 @@
+from app.shared.tasks.follows import join_community, leave_community
+from app.shared.tasks.likes import vote_for_post, vote_for_reply
+from app.shared.tasks.notes import make_reply, edit_reply
+from app.shared.tasks.deletes import delete_reply, restore_reply
+from app.shared.tasks.flags import report_reply
+
+from flask import current_app
+
+
+def task_selector(task_key, send_async=True, **kwargs):
+    tasks = {
+        'join_community': join_community,
+        'leave_community': leave_community,
+        'vote_for_post': vote_for_post,
+        'vote_for_reply': vote_for_reply,
+        'make_reply': make_reply,
+        'edit_reply': edit_reply,
+        'delete_reply': delete_reply,
+        'restore_reply': restore_reply,
+        'report_reply': report_reply
+    }
+
+    if current_app.debug:
+        send_async = False
+
+    if send_async:
+        tasks[task_key].delay(send_async=send_async, **kwargs)
+    else:
+        return tasks[task_key](send_async=send_async, **kwargs)
+

app/shared/tasks/deletes.py

@@ -0,0 +1,105 @@
+from app import celery
+from app.activitypub.signature import default_context, post_request
+from app.models import CommunityBan, PostReply, User
+from app.utils import gibberish, instance_banned
+
+from flask import current_app
+
+
+""" JSON format
+Delete:
+{
+  'id':
+  'type':
+  'actor':
+  'object':
+  '@context':
+  'audience':
+  'to': []
+  'cc': []
+}
+For Announce, remove @context from inner object, and use same fields except audience
+"""
+
+
+@celery.task
+def delete_reply(send_async, user_id, reply_id):
+    reply = PostReply.query.filter_by(id=reply_id).one()
+    delete_object(user_id, reply)
+
+
+@celery.task
+def restore_reply(send_async, user_id, reply_id):
+    reply = PostReply.query.filter_by(id=reply_id).one()
+    delete_object(user_id, reply, is_restore=True)
+
+
+def delete_object(user_id, object, is_restore=False):
+    user = User.query.filter_by(id=user_id).one()
+    community = object.community
+    if community.local_only or not community.instance.online():
+        return
+
+    banned = CommunityBan.query.filter_by(user_id=user_id, community_id=community.id).first()
+    if banned:
+        return
+    if not community.is_local():
+        if user.has_blocked_instance(community.instance.id) or instance_banned(community.instance.domain):
+            return
+
+    delete_id = f"https://{current_app.config['SERVER_NAME']}/activities/delete/{gibberish(15)}"
+    to = ["https://www.w3.org/ns/activitystreams#Public"]
+    cc = [community.public_url()]
+    delete = {
+      'id': delete_id,
+      'type': 'Delete',
+      'actor': user.public_url(),
+      'object': object.public_url(),
+      '@context': default_context(),
+      'audience': community.public_url(),
+      'to': to,
+      'cc': cc
+    }
+
+    if is_restore:
+        del delete['@context']
+        undo_id = f"https://{current_app.config['SERVER_NAME']}/activities/undo/{gibberish(15)}"
+        undo = {
+          'id': undo_id,
+          'type': 'Undo',
+          'actor': user.public_url(),
+          'object': delete,
+          '@context': default_context(),
+          'audience': community.public_url(),
+          'to': to,
+          'cc': cc
+        }
+
+    if community.is_local():
+        if is_restore:
+            del undo['@context']
+            object=undo
+        else:
+            del delete['@context']
+            object=delete
+
+        announce_id = f"https://{current_app.config['SERVER_NAME']}/activities/announce/{gibberish(15)}"
+        actor = community.public_url()
+        cc = [community.ap_followers_url]
+        announce = {
+          'id': announce_id,
+          'type': 'Announce',
+          'actor': actor,
+          'object': object,
+          '@context': default_context(),
+          'to': to,
+          'cc': cc
+        }
+        for instance in community.following_instances():
+            if instance.inbox and instance.online() and not user.has_blocked_instance(instance.id) and not instance_banned(instance.domain):
+               post_request(instance.inbox, announce, community.private_key, community.public_url() + '#main-key')
+    else:
+        payload = undo if is_restore else delete
+        post_request(community.ap_inbox_url, payload, user.private_key, user.public_url() + '#main-key')
+
+

app/shared/tasks/flags.py

@@ -0,0 +1,58 @@
+from app import celery
+from app.activitypub.signature import default_context, post_request
+from app.models import CommunityBan, PostReply, User
+from app.utils import gibberish, instance_banned
+
+from flask import current_app
+
+
+""" JSON format
+Flag:
+{
+  'id':
+  'type':
+  'actor':
+  'object':
+  '@context':
+  'audience':
+  'to': []
+  'summary':
+}
+"""
+
+
+@celery.task
+def report_reply(send_async, user_id, reply_id, summary):
+    reply = PostReply.query.filter_by(id=reply_id).one()
+    report_object(user_id, reply, summary)
+
+
+def report_object(user_id, object, summary):
+    user = User.query.filter_by(id=user_id).one()
+    community = object.community
+    if community.local_only or not community.instance.online():
+        return
+
+    banned = CommunityBan.query.filter_by(user_id=user_id, community_id=community.id).first()
+    if banned:
+        return
+    if not community.is_local():
+        if user.has_blocked_instance(community.instance.id) or instance_banned(community.instance.domain):
+            return
+
+    flag_id = f"https://{current_app.config['SERVER_NAME']}/activities/flag/{gibberish(15)}"
+    to = [community.public_url()]
+    flag = {
+      'id': flag_id,
+      'type': 'Flag',
+      'actor': user.public_url(),
+      'object': object.public_url(),
+      '@context': default_context(),
+      'audience': community.public_url(),
+      'to': to,
+      'summary': summary
+    }
+
+    post_request(community.ap_inbox_url, flag, user.private_key, user.public_url() + '#main-key')
+
+

app/shared/tasks/follows.py

@@ -0,0 +1,165 @@
+from app import cache, celery, db
+from app.activitypub.signature import default_context, post_request
+from app.models import Community, CommunityBan, CommunityJoinRequest, User
+from app.utils import community_membership, gibberish, joined_communities, instance_banned
+
+from flask import current_app, flash
+from flask_babel import _
+
+
+# would be in app/constants.py
+SRC_WEB = 1
+SRC_PUB = 2
+SRC_API = 3
+SRC_PLD = 4
+
+
+""" JSON format
+{
+  'id':
+  'type':
+  'actor':
+  'object':
+  '@context':   (outer object only)
+  'to': []
+}
+"""
+
+"""
+async:
+    delete memoized
+    add or delete community_join_request
+    used for admin preload in production (return values are ignored)
+    used for API
+sync:
+    add or delete community_member
+    used for debug mode
+    used for web users to provide feedback
+"""
+
+
+@celery.task
+def join_community(send_async, user_id, community_id, src):
+    user = User.query.filter_by(id=user_id).one()
+    community = Community.query.filter_by(id=community_id).one()
+
+    pre_load_message = {}
+    banned = CommunityBan.query.filter_by(user_id=user_id, community_id=community_id).first()
+    if banned:
+        if not send_async:
+            if src == SRC_WEB:
+                flash(_('You cannot join this community'))
+                return
+            elif src == SRC_PLD:
+                pre_load_message['user_banned'] = True
+                return pre_load_message
+            elif src == SRC_API:
+                raise Exception('banned_from_community')
+        return
+
+    if (not community.is_local() and
+        (user.has_blocked_instance(community.instance.id) or
+         instance_banned(community.instance.domain))):
+        if not send_async:
+            if src == SRC_WEB:
+                flash(_('Community is on banned or blocked instance'))
+                return
+            elif src == SRC_PLD:
+                pre_load_message['community_on_banned_or_blocked_instance'] = True
+                return pre_load_message
+            elif src == SRC_API:
+                raise Exception('community_on_banned_or_blocked_instance')
+        return
+
+    success = True
+    if not community.is_local() and community.instance.online():
+        join_request = CommunityJoinRequest(user_id=user_id, community_id=community_id)
+        db.session.add(join_request)
+        db.session.commit()
+
+        follow_id = f"https://{current_app.config['SERVER_NAME']}/activities/follow/{join_request.id}"
+        follow = {
+          'id': follow_id,
+          'type': 'Follow',
+          'actor': user.public_url(),
+          'object': community.public_url(),
+          '@context': default_context(),
+          'to': [community.public_url()],
+        }
+        success = post_request(community.ap_inbox_url, follow, user.private_key,
+                               user.public_url() + '#main-key', timeout=10)
+        if success is False or isinstance(success, str):
+            if not send_async:
+                db.session.query(CommunityJoinRequest).filter_by(user_id=user_id, community_id=community_id).delete()
+                db.session.commit()
+
+                if 'is not in allowlist' in success:
+                    msg_to_user = f'{community.instance.domain} does not allow us to join their communities.'
+                else:
+                    msg_to_user = "There was a problem while trying to communicate with remote server. Please try again later."
+
+                if src == SRC_WEB:
+                    flash(_(msg_to_user), 'error')
+                    return
+                elif src == SRC_PLD:
+                    pre_load_message['status'] = msg_to_user
+                    return pre_load_message
+                elif src == SRC_API:
+                    raise Exception(msg_to_user)
+
+    # for communities on local or offline instances, joining is instant
+    if success is True:
+        cache.delete_memoized(community_membership, user, community)
+        cache.delete_memoized(joined_communities, user.id)
+
+        if src == SRC_WEB:
+            flash('You joined ' + community.title)
+            return
+        elif src == SRC_PLD:
+            pre_load_message['status'] = 'joined'
+            return pre_load_message
+
+    return success
+
+
+@celery.task
+def leave_community(send_async, user_id, community_id):
+    user = User.query.filter_by(id=user_id).one()
+    community = Community.query.filter_by(id=community_id).one()
+
+    cache.delete_memoized(community_membership, user, community)
+    cache.delete_memoized(joined_communities, user.id)
+
+    if community.is_local():
+        return
+
+    join_request = CommunityJoinRequest.query.filter_by(user_id=user_id, community_id=community_id).one()
+    db.session.delete(join_request)
+    db.session.commit()
+
+    if (not community.instance.online() or
+       user.has_blocked_instance(community.instance.id) or
+       instance_banned(community.instance.domain)):
+        return
+
+    follow_id = f"https://{current_app.config['SERVER_NAME']}/activities/follow/{join_request.id}"
+    follow = {
+      'id': follow_id,
+      'type': 'Follow',
+      'actor': user.public_url(),
+      'object': community.public_url(),
+      'to': [community.public_url()]
+    }
+    undo_id = f"https://{current_app.config['SERVER_NAME']}/activities/undo/{gibberish(15)}"
+    undo = {
+      'id': undo_id,
+      'type': 'Undo',
+      'actor': user.public_url(),
+      'object': follow,
+      '@context': default_context(),
+      'to': [community.public_url()]
+    }
+
+    post_request(community.ap_inbox_url, undo, user.private_key, user.public_url() + '#main-key', timeout=10)
+
+

app/shared/tasks/likes.py

@@ -0,0 +1,108 @@
+from app import cache, celery
+from app.activitypub.signature import default_context, post_request
+from app.models import CommunityBan, Post, PostReply, User
+from app.utils import gibberish, instance_banned, recently_upvoted_posts, recently_downvoted_posts, recently_upvoted_post_replies, recently_downvoted_post_replies
+
+from flask import current_app
+
+
+""" JSON format
+{
+  'id':
+  'type':
+  'actor':
+  'object':
+  '@context':   (outer object only)
+  'audience':   (inner object only)
+  'to': []      (announce only)
+  'cc': []      (announce only)
+}
+"""
+
+@celery.task
+def vote_for_post(send_async, user_id, post_id, vote_to_undo, vote_direction):
+    post = Post.query.filter_by(id=post_id).one()
+    cache.delete_memoized(recently_upvoted_posts, user_id)
+    cache.delete_memoized(recently_downvoted_posts, user_id)
+    send_vote(user_id, post, vote_to_undo, vote_direction)
+
+
+@celery.task
+def vote_for_reply(send_async, user_id, reply_id, vote_to_undo, vote_direction):
+    reply = PostReply.query.filter_by(id=reply_id).one()
+    cache.delete_memoized(recently_upvoted_post_replies, user_id)
+    cache.delete_memoized(recently_downvoted_post_replies, user_id)
+    send_vote(user_id, reply, vote_to_undo, vote_direction)
+
+
+def send_vote(user_id, object, vote_to_undo, vote_direction):
+    user = User.query.filter_by(id=user_id).one()
+    community = object.community
+    if community.local_only or not community.instance.online():
+        return
+
+    banned = CommunityBan.query.filter_by(user_id=user_id, community_id=community.id).first()
+    if banned:
+        return
+    if not community.is_local():
+        if user.has_blocked_instance(community.instance.id) or instance_banned(community.instance.domain):
+            return
+
+    if vote_to_undo:
+        type=vote_to_undo
+    else:
+        type = 'Like' if vote_direction == 'upvote' else 'Dislike'
+    vote_id = f"https://{current_app.config['SERVER_NAME']}/activities/{type.lower()}/{gibberish(15)}"
+    actor = user.public_url(not(community.instance.votes_are_public() and user.vote_privately()))
+
+    vote = {
+      'id': vote_id,
+      'type': type,
+      'actor': actor,
+      'object': object.public_url(),
+      '@context': default_context(),
+      'audience': community.public_url()
+    }
+
+    if vote_to_undo:
+        del vote['@context']
+        undo_id = f"https://{current_app.config['SERVER_NAME']}/activities/undo/{gibberish(15)}"
+        undo = {
+          'id': undo_id,
+          'type': 'Undo',
+          'actor': actor,
+          'object': vote,
+          '@context': default_context(),
+          'audience': community.public_url()
+        }
+
+    if community.is_local():
+        if vote_to_undo:
+            del undo['@context']
+            object=undo
+        else:
+            del vote['@context']
+            object=vote
+
+        announce_id = f"https://{current_app.config['SERVER_NAME']}/activities/announce/{gibberish(15)}"
+        actor = community.public_url()
+        to = ["https://www.w3.org/ns/activitystreams#Public"]
+        cc = [community.ap_followers_url]
+        announce = {
+          'id': announce_id,
+          'type': 'Announce',
+          'actor': actor,
+          'object': object,
+          '@context': default_context(),
+          'to': to,
+          'cc': cc
+        }
+        for instance in community.following_instances():
+            if instance.inbox and instance.online() and not user.has_blocked_instance(instance.id) and not instance_banned(instance.domain):
+               post_request(instance.inbox, announce, community.private_key, community.public_url() + '#main-key')
+    else:
+        payload = undo if vote_to_undo else vote
+        post_request(community.ap_inbox_url, payload, user.private_key,
+                     user.public_url(not(community.instance.votes_are_public() and user.vote_privately())) + '#main-key')
+
+

app/shared/tasks/notes.py

@@ -0,0 +1,205 @@
+from app import cache, celery, db
+from app.activitypub.signature import default_context, post_request
+from app.models import Community, CommunityBan, CommunityJoinRequest, CommunityMember, Notification, Post, PostReply, User, utcnow
+from app.user.utils import search_for_user
+from app.utils import community_membership, gibberish, joined_communities, instance_banned, ap_datetime, \
+                      recently_upvoted_posts, recently_downvoted_posts, recently_upvoted_post_replies, recently_downvoted_post_replies
+
+from flask import current_app
+from flask_babel import _
+
+import re
+
+
+""" Reply JSON format
+{
+  'id':
+  'url':
+  'type':
+  'attributedTo':
+  'to': []
+  'cc': []
+  'tag': []
+  'audience':
+  'content':
+  'mediaType':
+  'source': {}
+  'inReplyTo':
+  'published':
+  'updated':        (inner oject of Update only)
+  'language': {}
+  'contentMap':{}
+  'distinguished'
+}
+"""
+""" Create / Update / Announce JSON format
+{
+  'id':
+  'type':
+  'actor':
+  'object':
+  'to': []
+  'cc': []
+  '@context':       (outer object only)
+  'audience':       (not in Announce)
+  'tag': []         (not in Announce)
+}
+"""
+
+
+
+@celery.task
+def make_reply(send_async, user_id, reply_id, parent_id):
+    send_reply(user_id, reply_id, parent_id)
+
+
+@celery.task
+def edit_reply(send_async, user_id, reply_id, parent_id):
+    send_reply(user_id, reply_id, parent_id, edit=True)
+
+
+def send_reply(user_id, reply_id, parent_id, edit=False):
+    user = User.query.filter_by(id=user_id).one()
+    reply = PostReply.query.filter_by(id=reply_id).one()
+    if parent_id:
+        parent = PostReply.query.filter_by(id=parent_id).one()
+    else:
+        parent = reply.post
+    community = reply.community
+
+    recipients = [parent.author]
+    pattern = r"@([a-zA-Z0-9_.-]*)@([a-zA-Z0-9_.-]*)\b"
+    matches = re.finditer(pattern, reply.body)
+    for match in matches:
+        recipient = None
+        if match.group(2) == current_app.config['SERVER_NAME']:
+            user_name = match.group(1)
+            try:
+                recipient = search_for_user(user_name)
+            except:
+                pass
+        else:
+            ap_id = f"{match.group(1)}@{match.group(2)}"
+            try:
+                recipient = search_for_user(ap_id)
+            except:
+                pass
+        if recipient:
+            add_recipient = True
+            for existing_recipient in recipients:
+                if ((not recipient.ap_id and recipient.user_name == existing_recipient.user_name) or
+                    (recipient.ap_id and recipient.ap_id == existing_recipient.ap_id)):
+                    add_recipient = False
+                    break
+            if add_recipient:
+                recipients.append(recipient)
+
+    if community.local_only:
+        for recipient in recipients:
+            if recipient.is_local() and recipient.id != parent.author.id:
+                already_notified = cache.get(f'{recipient.id} notified of {reply.id}')
+                if not already_notified:
+                    cache.set(f'{recipient.id} notified of {reply.id}', True, timeout=86400)
+                    notification = Notification(user_id=recipient.id, title=_('You have been mentioned in a comment'),
+                                        url=f"https://{current_app.config['SERVER_NAME']}/comment/{reply.id}",
+                                        author_id=user.id)
+                    recipient.unread_notifications += 1
+                    db.session.add(notification)
+                    db.session.commit()
+
+    if community.local_only or not community.instance.online():
+        return
+
+    banned = CommunityBan.query.filter_by(user_id=user_id, community_id=community.id).first()
+    if banned:
+        return
+    if not community.is_local():
+        if user.has_blocked_instance(community.instance.id) or instance_banned(community.instance.domain):
+            return
+
+    to = ["https://www.w3.org/ns/activitystreams#Public"]
+    cc = [community.public_url()]
+    tag = []
+    for recipient in recipients:
+        tag.append({'href': recipient.public_url(), 'name': recipient.mention_tag(), 'type': 'Mention'})
+        cc.append(recipient.public_url())
+    language = {'identifier': reply.language_code(), 'name': reply.language_name()}
+    content_map = {reply.language_code(): reply.body_html}
+    source = {'content': reply.body, 'mediaType': 'text/markdown'}
+    note = {
+      'id': reply.public_url(),
+      'url': reply.public_url(),
+      'type': 'Note',
+      'attributedTo': user.public_url(),
+      'to': to,
+      'cc': cc,
+      'tag': tag,
+      'audience': community.public_url(),
+      'content': reply.body_html,
+      'mediaType': 'text/html',
+      'source': source,
+      'inReplyTo': parent.public_url(),
+      'published': ap_datetime(reply.posted_at),
+      'language': language,
+      'contentMap': content_map,
+      'distinguished': False,
+    }
+    if edit:
+        note['updated']: ap_datetime(utcnow())
+
+    activity = 'create' if not edit else 'update'
+    create_id = f"https://{current_app.config['SERVER_NAME']}/activities/{activity}/{gibberish(15)}"
+    type = 'Create' if not edit else 'Update'
+    create = {
+      'id': create_id,
+      'type': type,
+      'actor': user.public_url(),
+      'object': note,
+      'to': to,
+      'cc': cc,
+      '@context': default_context(),
+      'tag': tag
+    }
+
+    domains_sent_to = [current_app.config['SERVER_NAME']]
+
+    if community.is_local():
+        del create['@context']
+
+        announce_id = f"https://{current_app.config['SERVER_NAME']}/activities/announce/{gibberish(15)}"
+        actor = community.public_url()
+        cc = [community.ap_followers_url]
+        announce = {
+          'id': announce_id,
+          'type': 'Announce',
+          'actor': community.public_url(),
+          'object': create,
+          'to': to,
+          'cc': cc,
+          '@context': default_context()
+        }
+        for instance in community.following_instances():
+            if instance.inbox and instance.online() and not user.has_blocked_instance(instance.id) and not instance_banned(instance.domain):
+                post_request(instance.inbox, announce, community.private_key, community.public_url() + '#main-key')
+                domains_sent_to.append(instance.domain)
+    else:
+        post_request(community.ap_inbox_url, create, user.private_key, user.public_url() + '#main-key')
+        domains_sent_to.append(community.instance.domain)
+
+    # send copy to anyone else Mentioned in reply. (mostly for other local users and users on microblog sites)
+    for recipient in recipients:
+        if recipient.instance.domain not in domains_sent_to:
+            post_request(recipient.instance.inbox, create, user.private_key, user.public_url() + '#main-key')
+        if recipient.is_local() and recipient.id != parent.author.id:
+            already_notified = cache.get(f'{recipient.id} notified of {reply.id}')
+            if not already_notified:
+                cache.set(f'{recipient.id} notified of {reply.id}', True, timeout=86400)
+                notification = Notification(user_id=recipient.id, title=_('You have been mentioned in a comment'),
+                                        url=f"https://{current_app.config['SERVER_NAME']}/comment/{reply.id}",
+                                        author_id=user.id)
+                recipient.unread_notifications += 1
+                db.session.add(notification)
+                db.session.commit()
+
+
+

app/static/js/scripts.js

@@ -33,8 +33,20 @@ document.addEventListener("DOMContentLoaded", function () {
     setupShowElementLinks();
     setupLightboxTeaser();
     setupLightboxPostBody();
+    setupPostTeaserHandler();
 });
 
+function setupPostTeaserHandler() {
+    document.querySelectorAll('.post_teaser_clickable').forEach(div => {
+        div.onclick = function() {
+            const firstAnchor = this.parentElement.querySelector('a');
+            if (firstAnchor) {
+                window.location.href = firstAnchor.href;
+            }
+        };
+    });
+}
+
 function setupYouTubeLazyLoad() {
     const lazyVideos = document.querySelectorAll(".video-wrapper");
 

app/static/scss/_typography.scss

@@ -166,7 +166,6 @@
 }
 
 .fe-reply {
-  margin-right: 2px;
   margin-top: -1px;
 }
 

app/static/styles.css

@@ -197,7 +197,6 @@
 }
 
 .fe-reply {
-  margin-right: 2px;
   margin-top: -1px;
 }
 
@@ -783,11 +782,9 @@ div.navbar {
   max-width: 78%;
 }
 .post_list .post_teaser .col_thumbnail {
+  float: right;
   width: 70px;
   position: relative;
-  padding-left: 0;
-  padding-right: 0;
-  left: -13px;
 }
 @media (min-width: 992px) {
   .post_list .post_teaser .col_thumbnail {
@@ -796,13 +793,18 @@ div.navbar {
 }
 .post_list .post_teaser .thumbnail {
   text-align: center;
-  height: 100%;
   align-content: center;
   display: flex;
+  height: 60px;
+  width: 60px;
+  overflow: hidden;
 }
 @media (min-width: 992px) {
   .post_list .post_teaser .thumbnail {
     align-items: center;
+    height: 90px;
+    width: 170px;
+    max-width: 100%;
   }
 }
 .post_list .post_teaser .thumbnail a {
@@ -862,10 +864,12 @@ div.navbar {
 .post_list .post_teaser.blocked .voting_buttons .upvote_button, .post_list .post_teaser.blocked .voting_buttons .downvote_button {
   font-size: 80%;
 }
+.post_list .post_teaser .post_teaser_clickable {
+  cursor: pointer;
+}
 
 .post_teaser_body {
   position: relative;
-  padding-bottom: 32px;
 }
 
 .post_teaser_article_preview, .post_teaser_link_preview {
@@ -889,7 +893,6 @@ div.navbar {
 .post_teaser_image_preview {
   text-align: center;
   margin-top: 10px;
-  margin-bottom: 10px;
 }
 .post_teaser_image_preview a {
   display: inline-block;
@@ -897,6 +900,7 @@ div.navbar {
 }
 .post_teaser_image_preview img {
   max-width: 100%;
+  min-width: 150px;
   margin-right: 4px;
   border-radius: 5px;
   height: auto;
@@ -905,7 +909,6 @@ div.navbar {
 .post_teaser_video_preview {
   text-align: center;
   margin-top: 10px;
-  margin-bottom: 10px;
   position: relative;
 }
 .post_teaser_video_preview .fe-video {
@@ -948,7 +951,6 @@ time {
 
 .author {
   padding-left: 3px;
-  display: inline-block;
 }
 .author a {
   border-left: 0;
@@ -960,12 +962,6 @@ time {
   width: 100%;
 }
 
-.post_utilities_bar_wrapper {
-  position: absolute;
-  bottom: -9px;
-  width: 97%;
-}
-
 .post_utilities_bar {
   display: flex;
   min-height: 44px;
@@ -977,38 +973,16 @@ time {
     justify-content: left;
   }
 }
-.post_utilities_bar .cross_post_button {
+.post_utilities_bar div {
   display: flex;
-  justify-content: left;
+  justify-content: center;
   align-items: center;
-  width: 44px;
-  height: 44px;
-  line-height: 44px;
-  padding: 3px;
-  padding-left: 0;
-}
-.post_utilities_bar .post_replies_link {
-  display: flex;
   min-width: 44px;
   height: 44px;
-  padding: 3px;
+  line-height: 44px;
-  align-items: center;
-  margin-left: -3px;
 }
-.post_utilities_bar .voting_buttons_new {
+.post_utilities_bar .notify_toggle {
-  display: flex;
+  margin-left: auto; /* pull right */
-  height: 44px;
-  padding: 3px;
-  justify-content: center;
-  align-items: center;
-}
-.post_utilities_bar .post_cross_post_link, .post_utilities_bar .post_options_link, .post_utilities_bar .preview_image {
-  display: flex;
-  width: 44px;
-  height: 44px;
-  padding: 3px;
-  justify-content: center;
-  align-items: center;
 }
 
 .post_full .post_utilities_bar .voting_buttons_new {
@@ -1204,9 +1178,6 @@ time {
   height: auto;
 }
 
-.voting_buttons_new {
-  display: inline-block;
-}
 .voting_buttons_new .upvote_button, .voting_buttons_new .downvote_button {
   position: relative; /* so the htmx-indicators can be position: absolute */
   display: inline-block;
@@ -1230,10 +1201,14 @@ time {
 }
 .voting_buttons_new .upvote_button.voted_up, .voting_buttons_new .downvote_button.voted_up {
   color: green;
+}
+.voting_buttons_new .upvote_button.voted_up .fe, .voting_buttons_new .downvote_button.voted_up .fe {
   font-weight: bold;
 }
 .voting_buttons_new .upvote_button.voted_down, .voting_buttons_new .downvote_button.voted_down {
   color: darkred;
+}
+.voting_buttons_new .upvote_button.voted_down .fe, .voting_buttons_new .downvote_button.voted_down .fe {
   font-weight: bold;
 }
 .voting_buttons_new .upvote_button .htmx-indicator {
@@ -1335,6 +1310,7 @@ time {
 .comment {
   clear: both;
   margin-left: 15px;
+  padding-left: 0px;
   padding-top: 8px;
 }
 .comment .limit_height {
@@ -1386,26 +1362,27 @@ time {
 }
 .comment .comment_actions {
   margin-top: -18px;
-  position: relative;
+  display: flex;
-  padding-bottom: 5px;
+  min-height: 44px;
+  /* justify-content: space-between; */
+  justify-content: left;
 }
-.comment .comment_actions a {
+@media (min-width: 992px) {
-  text-decoration: none;
+  .comment .comment_actions {
-  padding: 0;
+    justify-content: left;
+  }
 }
-.comment .comment_actions .hide_button {
+.comment .comment_actions div {
-  display: inline-block;
+  display: flex;
-}
+  justify-content: center;
-.comment .comment_actions .hide_button a {
+  align-items: center;
-  padding: 5px 15px;
+  min-width: 44px;
+  height: 44px;
+  line-height: 44px;
 }
 .comment .comment_actions .notify_toggle {
-  display: inline-block;
+  margin-left: auto; /* pull right */
-}
-.comment .comment_actions .notify_toggle a {
-  text-decoration: none;
   font-size: 87%;
-  padding: 5px 15px;
 }
 .comment .replies {
   margin-top: 0;
@@ -1601,6 +1578,11 @@ h1 .warning_badge {
   height: auto;
 }
 
+.wiki_page img {
+  max-width: 100%;
+  height: auto;
+}
+
 #post_reply_markdown_editor_enabler {
   display: none;
   position: absolute;
@@ -1801,6 +1783,9 @@ form h5 {
   position: relative;
   top: -11px;
 }
+.coolfieldset legend.tweak-top {
+  top: -18px;
+}
 
 .coolfieldset legend, .coolfieldset.expanded legend {
   background: whitesmoke url(/static/images/expanded.gif) no-repeat center left;
@@ -1865,30 +1850,6 @@ form h5 {
   margin-bottom: 10px;
 }
 
-.post_options_link {
-  display: block;
-  width: 41px;
-  text-decoration: none;
-}
-
-.comment_actions_link {
-  display: block;
-  position: absolute;
-  top: 3px;
-  right: -16px;
-  width: 41px;
-  text-decoration: none;
-}
-
-.notify_toggle {
-  display: block;
-  position: absolute;
-  top: 2px;
-  right: 30px;
-  width: 41px;
-  text-decoration: none;
-}
-
 .alert {
   width: 96%;
 }
@@ -1977,6 +1938,10 @@ form h5 {
   margin-bottom: 0;
 }
 
+.pt-05 {
+  padding-top: 0.12rem !important;
+}
+
 /* high contrast */
 @media (prefers-contrast: more) {
   :root {

app/static/styles.scss

@@ -369,11 +369,9 @@ div.navbar {
     }
 
     .col_thumbnail {
+      float: right;
       width: 70px;
       position: relative;
-      padding-left: 0;
-      padding-right: 0;
-      left: -13px;
       @include breakpoint(tablet) {
         width: 170px;
       }
@@ -382,11 +380,16 @@ div.navbar {
     .thumbnail {
 
       text-align: center;
-      height: 100%;
       align-content: center;
       display: flex;
+      height: 60px;
+      width: 60px;
+      overflow: hidden;
       @include breakpoint(tablet) {
         align-items: center;
+        height: 90px;
+        width: 170px;
+        max-width: 100%;
       }
 
       a {
@@ -455,12 +458,15 @@ div.navbar {
         }
       }
     }
+
+    .post_teaser_clickable {
+      cursor: pointer;
+    }
   }
 }
 
 .post_teaser_body {
   position: relative;
-  padding-bottom: 32px;
 }
 
 .post_teaser_article_preview, .post_teaser_link_preview {
@@ -485,13 +491,13 @@ div.navbar {
 .post_teaser_image_preview {
   text-align: center;
   margin-top: 10px;
-  margin-bottom: 10px;
   a {
     display: inline-block;
     max-width: 512px;
   }
   img {
     max-width: 100%;
+    min-width: 150px;
     margin-right: 4px;
     border-radius: 5px;
     height: auto;
@@ -501,7 +507,6 @@ div.navbar {
 .post_teaser_video_preview {
   text-align: center;
   margin-top: 10px;
-  margin-bottom: 10px;
   position: relative;
 
   .fe-video {
@@ -548,7 +553,6 @@ time {
 
 .author {
   padding-left: 3px;
-  display: inline-block;
 
   a {
     border-left: 0;
@@ -561,12 +565,6 @@ time {
   width: 100%;
 }
 
-.post_utilities_bar_wrapper {
-  position: absolute;
-  bottom: -9px;
-  width: 97%;
-}
-
 .post_utilities_bar {
   display: flex;
   min-height: $min-touch-target;
@@ -576,41 +574,17 @@ time {
     justify-content: left;
   }
 
-  .cross_post_button {
+  div {
     display: flex;
-    justify-content: left;
+    justify-content: center;
     align-items: center;
-    width: $min-touch-target;
-    height: $min-touch-target;
-    line-height: $min-touch-target;
-    padding: 3px;
-    padding-left: 0;
-  }
-
-  .post_replies_link {
-    display: flex;
     min-width: $min-touch-target;
     height: $min-touch-target;
-    padding: 3px;
+    line-height: $min-touch-target;
-    align-items: center;
-    margin-left: -3px;
   }
 
-  .voting_buttons_new {
+  .notify_toggle {
-    display: flex;
+    margin-left: auto; /* pull right */
-    height: $min-touch-target;
-    padding: 3px;
-    justify-content: center;
-    align-items: center;
-  }
-
-   .post_cross_post_link, .post_options_link, .preview_image {
-    display: flex;
-    width: $min-touch-target;
-    height: $min-touch-target;
-    padding: 3px;
-    justify-content: center;
-    align-items: center;
   }
 }
 
@@ -838,8 +812,6 @@ time {
 }
 
 .voting_buttons_new {
-  display: inline-block;
-
   .upvote_button, .downvote_button {
     position: relative; /* so the htmx-indicators can be position: absolute */
     display: inline-block;
@@ -866,11 +838,15 @@ time {
 
     &.voted_up {
       color: green;
-      font-weight: bold;
+      .fe {
+        font-weight: bold;
+      }
     }
     &.voted_down {
       color: darkred;
-      font-weight: bold;
+      .fe {
+        font-weight: bold;
+      }
     }
   }
 
@@ -988,6 +964,7 @@ time {
 .comment {
   clear: both;
   margin-left: 15px;
+  padding-left: 0px;
   padding-top: 8px;
 
   .limit_height {
@@ -1048,28 +1025,26 @@ time {
 
   .comment_actions {
     margin-top: -18px;
-    position: relative;
+    display: flex;
-    padding-bottom: 5px;
+    min-height: $min-touch-target;
-    a {
+    /* justify-content: space-between; */
-      text-decoration: none;
+    justify-content: left;
-      padding: 0;
+    @include breakpoint(tablet) {
+      justify-content: left;
     }
 
-    .hide_button {
+    div {
-      display: inline-block;
+      display: flex;
-
+      justify-content: center;
-      a {
+      align-items: center;
-        padding: 5px 15px;
+      min-width: $min-touch-target;
-      }
+      height: $min-touch-target;
+      line-height: $min-touch-target;
     }
 
     .notify_toggle {
-      display: inline-block;
+      margin-left: auto; /* pull right */
-      a {
+      font-size: 87%;
-        text-decoration: none;
-        font-size: 87%;
-        padding: 5px 15px;
-      }
     }
   }
 
@@ -1278,6 +1253,13 @@ h1 .warning_badge {
   }
 }
 
+.wiki_page {
+  img {
+    max-width: 100%;
+    height: auto;
+  }
+}
+
 #post_reply_markdown_editor_enabler {
   display: none;
   position: absolute;
@@ -1491,6 +1473,10 @@ form {
   display: block;
   position: relative;
   top: -11px;
+
+  &.tweak-top {
+    top: -18px;
+  }
 }
 
 .coolfieldset legend, .coolfieldset.expanded legend{
@@ -1572,30 +1558,6 @@ form {
   margin-bottom: 10px;
 }
 
-.post_options_link {
-  display: block;
-  width: 41px;
-  text-decoration: none;
-}
-
-.comment_actions_link {
-  display: block;
-  position: absolute;
-  top: 3px;
-  right: -16px;
-  width: 41px;
-  text-decoration: none;
-}
-
-.notify_toggle  {
-  display: block;
-  position: absolute;
-  top: 2px;
-  right: 30px;
-  width: 41px;
-  text-decoration: none;
-}
-
 .alert {
   width: 96%;
 }
@@ -1687,6 +1649,10 @@ form {
   }
 }
 
+.pt-05 {
+  padding-top: .12rem !important;
+}
+
 
 /* high contrast */
 @import "scss/high_contrast";

app/templates/_side_pane.html

@@ -32,7 +32,7 @@
                  <h2>{{ community.title }}</h2>
             </div>
             <div class="card-body">
-                <p>{{ community.description_html|safe if community.description_html else '' }}</p>
+                <p>{{ community.description_html|community_links|safe if community.description_html else '' }}</p>
                 <p>{{ community.rules_html|safe if community.rules_html else '' }}</p>
                 {% if len(mods) > 0 and not community.private_mods -%}
                     <h3>Moderators</h3>
@@ -45,7 +45,7 @@
                     <p class="red small">{{ _('Moderators have not been active recently.') }}</p>
                     {% endif -%}
                 {% endif -%}
-                {% if not community.is_local() -%}
+                {% if rss_feed and not community.is_local() -%}
                     <ul>
                         <li><p><a href="{{ community.public_url() }}">{{ _('View community on original server') }}</a></p></li>
                         <li><p><a href="{{ url_for('community.retrieve_remote_post', community_id=community.id) }}">{{ _('Retrieve a post from the original server') }}</a></p></li>

app/templates/about.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 
 {% block app_content %}
 <div class="row">

app/templates/admin/activities.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_activities' %}
 

app/templates/admin/activity_json.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_activities' %}
 

app/templates/admin/add_user.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 {% set active_child = 'admin_users' %}
 

app/templates/admin/approve_registrations.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_approve_registrations' %}
 

app/templates/admin/communities.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_communities' %}
 
@@ -19,12 +19,10 @@
         <table class="table table-striped">
             <tr>
                <th>Name</th>
-               <th>Title</th>
                <th>Topic</th>
                <th>#&nbsp;Posts</th>
                <th>Retention</th>
                <th>Layout</th>
-               <th title="{{ _('Posts show on home page.') }}">Home</th>
                <th title="{{ _('Posts can be popular.') }}">Popular</th>
                <th title="{{ _('Posts show in the All feed.') }}">All</th>
                <th title="{{ _('Content warning, NSFW or NSFL set for community.') }}">Warning</th>
@@ -32,13 +30,12 @@
             </tr>
             {% for community in communities.items %}
                 <tr>
-                    <td><a href="/c/{{ community.link() }}">{{ community.name }}</a></td>
+                    <td>{{ render_communityname(community, add_domain=False) }}{% if community.banned %} (banned){% endif %}<br />
-                    <td>{{ render_communityname(community) }}{% if community.banned %} (banned){% endif %}</td>
+                    !<a href="/c/{{ community.link() }}">{{ community.name }}</a><wbr />@<a href="{{ community.ap_profile_id }}">{{ community.ap_domain }}</a></td>
                     <td>{{ community.topic.name  }}</td>
                     <td>{{ community.post_count  }}</td>
                     <td>{{ community.content_retention if community.content_retention != -1 }}</td>
                     <td>{{ community.default_layout if community.default_layout }}</td>
-                    <th>{{ '&check;'|safe if community.show_home else '&cross;'|safe }}</th>
                     <th>{{ '&check;'|safe if community.show_popular else '&cross;'|safe }}</th>
                     <th>{{ '&check;'|safe if community.show_all else '&cross;'|safe }}</th>
                     <th>{{ '&#x26A0;'|safe if community.nsfw or community.nsfl or community.content_warning else ''|safe }}</th>

app/templates/admin/deleted_posts.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_content_deleted' %}
 
@@ -33,7 +33,7 @@
             <h2 class="mt-4" id="comments">Deleted comments</h2>
             <div class="post_list">
                 {% for post_reply in post_replies.items %}
-                    {% with teaser=True, disable_voting=True, no_collapse=True %}
+                    {% with teaser=True, disable_voting=True, no_collapse=True, show_deleted=True %}
                         {% include 'post/_post_reply_teaser.html' %}
                     {% endwith %}
                     <hr />

app/templates/admin/edit_community.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 {% set active_child = 'admin_communities' %}
 
@@ -41,7 +41,6 @@
                 {{ render_field(form.banned) }}
                 {{ render_field(form.local_only) }}
                 {{ render_field(form.new_mods_wanted) }}
-                {{ render_field(form.show_home) }}
                 {{ render_field(form.show_popular) }}
                 {{ render_field(form.show_all) }}
                 {{ render_field(form.low_quality) }}

app/templates/admin/edit_instance.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_instances' %}
 

app/templates/admin/edit_topic.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_topics' %}
 

app/templates/admin/edit_user.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 {% set active_child = 'admin_users' %}
 
@@ -26,6 +26,8 @@
             {{ render_field(form.bot) }}
             {{ render_field(form.verified) }}
             {{ render_field(form.banned) }}
+            {{ render_field(form.ban_posts) }}
+            {{ render_field(form.ban_comments) }}
             <p>receive newsletter: {{ user.newsletter }}</p>
             {{ render_field(form.hide_nsfw) }}
             {{ render_field(form.hide_nsfl) }}

app/templates/admin/federation.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form, render_field %}
 {% set active_child = 'admin_federation' %}
 
@@ -17,7 +17,8 @@
 <hr />
 <div class="row">
     <div class="col">
-        <p>Import / Export Bans</p>
+        <h4>{{ _('Import / Export Bans') }}</h4>
+        <p>Use this to import or export banned instances, domains, tags, and / or users.</p>
         <p>JSON format:</p>
         <pre><code>
             {
@@ -36,15 +37,32 @@
     </div>
 </div>
 <hr />
-<div class="row">
+<fieldset class="coolfieldset border mt-4 p-2 pb-3 mb-4">
-    <div class="column">
+    <legend class="tweak-top">Bulk community import</legend>
-        <p>Use this to "pre-load" known threadiverse communities, as ranked by posts and activity. The list of communities pulls from the same list as <a href="https://lemmyverse.net/communities">LemmyVerse</a>. NSFW communities and communities from banned instances are excluded.</p>
+        <div class="row">
-        {% if current_app_debug %}
+            <div class="column">
-        <p>*** This instance is in development mode. Loading more than 6 communities here could cause timeouts, depending on how your networking is setup. ***</p>
+                <h4>{{ _('Remote server scan') }}</h4>
-        {% endif %}
+                <p>{{ _('Use this to scan a remote lemmy server and "pre-load" it\'s communities, as ranked by posts and activity. NSFW communities and communities from banned instances are excluded.') }}</p>
-        {{ render_form(preload_form) }}
+                <p>{{ _('Input should be in the form of <strong>https://server-name.tld</strong>') }}</p>
-    </div>
+                {% if current_app_debug %}
-</div>
+                <p>*** This instance is in development mode. This function could cause timeouts depending on how your networking is setup. ***</p>
+                {% endif %}
+                {{ render_form(remote_scan_form) }}
+            </div>
+        </div>
+        <hr />
+
+        <div class="row">
+            <div class="column">
+                <h4>{{ _('Load communities from Lemmyverse data') }}</h4>
+                <p>{{ _('Use this to "pre-load" known threadiverse communities, as ranked by posts and activity. The list of communities pulls from the same list as <a href="https://lemmyverse.net/communities">LemmyVerse</a>. NSFW communities and communities from banned instances are excluded. Communities with less than 100 posts and less than 500 active users in the past week are excluded.') }}</p>
+                {% if current_app_debug %}
+                <p>*** This instance is in development mode. This function could cause timeouts depending on how your networking is setup. ***</p>
+                {% endif %}
+                {{ render_form(preload_form) }}
+            </div>
+        </div>
+</fieldset>
 <hr />
 <div class="row">
     <div class="col">

app/templates/admin/home.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/admin/misc.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_misc' %}
 

app/templates/admin/permissions.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_permissions' %}
 

app/templates/admin/posts.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_content_trash' %}
 

app/templates/admin/reports.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_reports' %}
 

app/templates/admin/site.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 {% set active_child = 'admin_site' %}
 

app/templates/admin/spam_posts.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_content_spam' %}
 
@@ -19,7 +19,7 @@
             <h2 class="mt-4" id="comments">Downvoted comments</h2>
             <div class="post_list">
                 {% for post_reply in post_replies.items %}
-                    {% with teaser=True, disable_voting=True, no_collapse=True %}
+                    {% with teaser=True, disable_voting=True, no_collapse=True, show_deleted=True %}
                         {% include 'post/_post_reply_teaser.html' %}
                     {% endwith %}
                     <hr />

app/templates/admin/topics.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_topics' %}
 

app/templates/admin/users.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_users' %}
 
@@ -20,7 +20,6 @@
         <table class="table table-striped mt-1">
             <tr>
                <th title="{{ _('Display name.') }}">{{ _('Name') }}</th>
-               <th>{{ _('Local/Remote') }}</th>
                <th title="{{ _('Last seen.') }}">{{ _('Seen') }}</th>
                <th title="{{ _('Attitude: Percentage of up votes vs. down votes the account made.') }}">{{ _('Attitude') }}</th>
                <th title="{{ _('Reputation: The Karma of the account. Total up votes minus down votes they got.') }}">{{ _('Reputation') }}</th>
@@ -32,29 +31,23 @@
             </tr>
             {% for user in users.items %}
                 <tr>
-                    <td><a href="/u/{{ user.link() }}">
+                    <td>{{ render_username(user, add_domain=False) }}<br />
-                            <img src="{{ user.avatar_thumbnail() }}" class="community_icon rounded-circle" loading="lazy" />
+                    <a href="/u/{{ user.link() }}">{{ user.user_name }}</a>{% if not user.is_local() %}<wbr />@<a href="{{ user.ap_profile_id }}">{{ user.ap_domain }}</a>{% endif %}</td>
-                            {{ user.display_name() }}</a></td>
-                    <td>{% if user.is_local() %}Local{% else %}<a href="{{ user.ap_profile_id }}">Remote</a>{% endif %}</td>
                     <td>{% if request.args.get('local_remote', '') == 'local' %}
                             {{ arrow.get(user.last_seen).humanize(locale=locale) }}
                         {% else %}
                             {{ user.last_seen }}
                         {% endif %}
                     </td>
-                    <td>{{ user.attitude * 100 if user.attitude != 1 }}</td>
+                    <td>{% if user.attitude != 1 %}{{ (user.attitude * 100) | round | int }}%{% endif %}</td>
-                    <td>{{ 'R ' + str(user.reputation) if user.reputation }}</td>
+                    <td>{% if user.reputation %}R {{ user.reputation | round | int }}{% endif %}</td>
-                    <td>{{ '<span class="red">Banned</span>'|safe if user.banned }} </td>
+                    <td>{{ '<span class="red">Banned</span>'|safe if user.banned }}
+                        {{ '<span class="red">Banned posts</span>'|safe if user.ban_posts }}
+                        {{ '<span class="red">Banned comments</span>'|safe if user.ban_comments }}</td>
                     <td>{{ user.reports if user.reports > 0 }} </td>
                     <td>{{ user.ip_address if user.ip_address }}<br />{{ user.ip_address_country if user.ip_address_country }}</td>
                     <td>{{ user.referrer if user.referrer }} </td>
-                    <td><a href="/u/{{ user.link() }}">View local</a> |
+                    <td><a href="{{ url_for('admin.admin_user_edit', user_id=user.id) }}">Edit</a> |
-                        {% if not user.is_local() %}
-                            <a href="{{ user.ap_profile_id }}">View remote</a> |
-                        {% else %}
-                            View remote |
-                        {% endif %}
-                        <a href="{{ url_for('admin.admin_user_edit', user_id=user.id) }}">Edit</a> |
                         <a href="{{ url_for('admin.admin_user_delete', user_id=user.id) }}" class="confirm_first">Delete</a>
                     </td>
                 </tr>

app/templates/admin/users_trash.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'admin_users_trash' %}
 
@@ -22,7 +22,6 @@
         <table class="table table-striped mt-1">
             <tr>
                <th>Name</th>
-               <th>Local/Remote</th>
                <th>Seen</th>
                <th>Attitude</th>
                <th>Rep</th>
@@ -34,29 +33,21 @@
             </tr>
             {% for user in users.items %}
                 <tr>
-                    <td><a href="/u/{{ user.link() }}">
+                    <td>{{ render_username(user, add_domain=False) }}<br />
-                            <img src="{{ user.avatar_thumbnail() }}" class="community_icon rounded-circle" loading="lazy" />
+                    <a href="/u/{{ user.link() }}">{{ user.user_name }}</a>{% if not user.is_local() %}<wbr />@<a href="{{ user.ap_profile_id }}">{{ user.ap_domain }}</a>{% endif %}</td>
-                            {{ user.display_name() }}</a></td>
-                    <td>{% if user.is_local() %}Local{% else %}<a href="{{ user.ap_profile_id }}">Remote</a>{% endif %}</td>
                     <td>{% if request.args.get('local_remote', '') == 'local' %}
                             {{ arrow.get(user.last_seen).humanize(locale=locale) }}
                         {% else %}
                             {{ user.last_seen }}
                         {% endif %}
                     </td>
-                    <td>{{ user.attitude * 100 if user.attitude != 1 }}</td>
+                    <td>{% if user.attitude != 1 %}{{ (user.attitude * 100) | round | int }}%{% endif %}</td>
-                    <td>{{ 'R ' + str(user.reputation) if user.reputation }}</td>
+                    <td>{% if user.reputation %}R {{ user.reputation | round | int }}{% endif %}</td>
                     <td>{{ '<span class="red">Banned</span>'|safe if user.banned }} </td>
                     <td>{{ user.reports if user.reports > 0 }} </td>
                     <td>{{ user.ip_address if user.ip_address }} </td>
                     <td>{{ user.referrer if user.referrer }} </td>
-                    <td><a href="/u/{{ user.link() }}">View local</a> |
+                    <td><a href="{{ url_for('admin.admin_user_edit', user_id=user.id) }}">Edit</a> |
-                        {% if not user.is_local() %}
-                            <a href="{{ user.ap_profile_id }}">View remote</a> |
-                        {% else %}
-                            View remote |
-                        {% endif %}
-                        <a href="{{ url_for('admin.admin_user_edit', user_id=user.id) }}">Edit</a> |
                         <a href="{{ url_for('admin.admin_user_delete', user_id=user.id) }}" class="confirm_first">Delete</a>
                     </td>
                 </tr>

app/templates/auth/register.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block scripts %}

app/templates/auth/reset_password.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/auth/reset_password_request.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/base.html

@@ -1,13 +1,17 @@
-{% macro render_username(user) -%}
+{% macro render_username(user, add_domain=True) -%}
 <span class="render_username">
     {% if user.deleted -%}
-        [deleted]
+        {% if current_user.is_authenticated and current_user.is_admin() -%}
+            <a href="/u/{{ user.link() }}" title="{{ user.ap_id if user.ap_id != none else user.user_name }}" aria-label="{{ _('Author') }}">[deleted]</a>
+        {% else -%}
+            [deleted]
+        {% endif -%}
     {% else -%}
         <a href="/u/{{ user.link() }}" title="{{ user.ap_id if user.ap_id != none else user.user_name }}" aria-label="{{ _('Author') }}">
             {% if user.avatar_id and not low_bandwidth and not collapsed -%}
                 <img src="{{ user.avatar_thumbnail() }}" alt="" loading="lazy" />
             {% endif -%}
-            {{ user.display_name() }}{% if not user.is_local() %}<span class="text-muted">@{{ user.ap_domain }}</span>{% endif %}
+            {{ user.display_name() }}{% if add_domain and not user.is_local() %}<span class="text-muted">@{{ user.ap_domain }}</span>{% endif %}
         </a>
         {% if user.created_recently() -%}
             <span class="fe fe-new-account" title="New account"> </span>
@@ -26,13 +30,13 @@
     {% endif -%}
 </span>
 {% endmacro -%}
-{% macro render_communityname(community) -%}
+{% macro render_communityname(community, add_domain=True) -%}
     <span class="render_community">
         <a href="/c/{{ community.link() }}" aria-label="{{ _('Go to community %(name)s', name=community.name) }}">
             {% if community.icon_id and not low_bandwidth and not collapsed -%}
                 <img src="{{ community.icon_image('tiny') }}" class="community_icon rounded-circle" alt="" loading="lazy" />
             {% endif -%}
-            c/{{ community.display_name() }}
+            {{ community.title }}{% if add_domain and not community.is_local() %}<span class="text-muted">@{{ community.ap_domain }}</span>{% endif %}
         </a>
     </span>
 {% endmacro -%}

app/templates/chat/blocked.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% set active_child = 'chats' %}
 
 {% block app_content %}

app/templates/chat/chat_options.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% set active_child = 'chats' %}
 {% from 'bootstrap/form.html' import render_form %}
 

app/templates/chat/conversation.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% set active_child = 'chats' %}
 {% from 'bootstrap/form.html' import render_form %}
 

app/templates/chat/denied.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'chats' %}
 

app/templates/chat/empty.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% set active_child = 'chats' %}
 
 {% block app_content %}

app/templates/chat/new_message.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% set active_child = 'chats' %}
 {% from 'bootstrap/form.html' import render_form %}
 

app/templates/chat/report.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% set active_child = 'chats' %}
 {% from 'bootstrap/form.html' import render_form %}
 

app/templates/community/add_local.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 
 {% block app_content %}
@@ -22,9 +22,9 @@
                         </div>
                         {{ render_field(form.description) }}
                         {{ render_field(form.icon_file) }}
-                        <small class="field_hint">Provide a square image that looks good when small.</small>
+                        <small class="field_hint">{{ _('Provide a square image that looks good when small. SVG is allowed.') }}</small>
                         {{ render_field(form.banner_file) }}
-                        <small class="field_hint">Provide a wide image - letterbox orientation.</small>
+                        <small class="field_hint">{{ _('Provide a wide image - letterbox orientation.') }}</small>
                         {{ render_field(form.rules) }}
                         {{ render_field(form.nsfw) }}
                         {{ render_field(form.local_only) }}

app/templates/community/add_remote.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/community/community_ban_user.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/community/community_delete.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/community/community_edit.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 
 {% block app_content %}

app/templates/community/community_find_moderator.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/community/community_mod_list.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 
 {% block app_content %}

app/templates/community/community_moderate.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 
 {% block app_content %}

app/templates/community/community_moderate_report_escalate.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/community/community_moderate_report_resolve.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/community/community_moderate_subscribers.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 
 {% block app_content %}

app/templates/community/community_modlog.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 
 {% block app_content %}

app/templates/community/community_wiki_edit.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form, render_field %}
 
 {% block app_content %}

app/templates/community/community_wiki_list.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 
 {% block app_content %}

app/templates/community/community_wiki_page_view.html

@@ -10,7 +10,7 @@
 <div class="row">
     <div class="col-12 col-md-8 position-relative main_pane">
         <div class="row position-relative">
-            <div class="col post_col post_type_normal">
+            <div class="col post_col post_type_normal wiki_page">
                 <nav aria-label="breadcrumb" id="breadcrumb_nav" title="Navigation">
                   <ol class="breadcrumb">
                     {% for breadcrumb in breadcrumbs -%}

app/templates/community/community_wiki_revisions.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_field %}
 
 {% block app_content %}

app/templates/community/lookup_remote.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/community/retrieve_remote_post.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/dev/tools.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 {% set active_child = 'dev_tools' %}
 

app/templates/domain/domain.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}
@@ -16,6 +16,9 @@
           </ol>
         </nav>
         <h1 class="mt-2">{{ domain.name }}</h1>
+        {% if domain.post_warning -%}
+            <p>{{ domain.post_warning }}</p>
+        {% endif -%}
         <div class="post_list">
             {% for post in posts.items %}
                 {% include 'post/_post_teaser.html' %}

app/templates/domain/domains.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/domain/domains_blocked.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/donate.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 
 {% block app_content %}
 <div class="row">

app/templates/errors/401.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 
 {% block app_content %}
     <div class="row">

app/templates/errors/404.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 
 {% block app_content %}
     <div class="row">

app/templates/errors/500.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 
 {% block app_content %}
     <div class="row">

app/templates/instance/list_instances.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/instance/overview.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/instance/people.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 {% from 'bootstrap/form.html' import render_form %}
 
 {% block app_content %}

app/templates/keyboard_shortcuts.html

@@ -2,7 +2,7 @@
     {% extends 'themes/' + theme() + '/base.html' %}
 {% else %}
     {% extends "base.html" %}
-{% endif %} %}
+{% endif %}
 
 {% block app_content %}
 <div class="row">

app/templates/list_communities.html

@@ -16,10 +16,14 @@
             <a href="/communities/local" aria-label="{{ _('Communities on this server') }}" class="btn {{ 'btn-primary' if request.path == '/communities/local' else 'btn-outline-secondary' }}">
                 {{ _('Local') }}
             </a>
+            {% if current_user.is_authenticated -%}
             <a href="/communities/subscribed" aria-label="{{ _('Joined communities') }}" class="btn {{ 'btn-primary' if request.path == '/communities/subscribed' else 'btn-outline-secondary' }}">
                 {{ _('Joined') }}
             </a>
-
+            <a href="/communities/notsubscribed" aria-label="{{ _('Not Joined communities') }}" class="btn {{ 'btn-primary' if request.path == '/communities/notsubscribed' else 'btn-outline-secondary' }}">
+                {{ _('Not Joined') }}
+            </a>
+            {% endif -%}
         </div>
     </div>
     <div class="col-auto">

app/templates/modlog.html

@@ -29,12 +29,12 @@
                             {% elif modlog_entry.link_text -%}
                                 {{ modlog_entry.link_text }}
                             {% endif -%}
-                            {% if modlog_entry.reason -%}
-                               <br>{{ _('Reason:') }} {{ modlog_entry.reason }}
-                            {% endif -%}
                             {% if modlog_entry.community_id -%}
                                 <a href="/c/{{ modlog_entry.community.link() }}">{{ _(' in %(community_name)s', community_name='' + modlog_entry.community.display_name()) }}</a>
                             {% endif -%}
+                            {% if modlog_entry.reason -%}
+                                <br>{{ _('Reason:') }} {{ modlog_entry.reason }}
+                            {% endif -%}
                         </td>
                     </tr>
                 {% endfor %}

app/templates/post/_breadcrumb_nav.html

@@ -0,0 +1,8 @@
+<nav aria-label="breadcrumb" id="breadcrumb_nav" title="Navigation">
+  <ol class="breadcrumb">
+    {% for breadcrumb in breadcrumbs -%}
+      <li class="breadcrumb-item">{% if breadcrumb.url -%}<a href="{{ breadcrumb.url }}">{% endif -%}{{ breadcrumb.text }}{% if breadcrumb.url -%}</a>{% endif -%}</li>
+    {% endfor -%}
+    <li class="breadcrumb-item"><a href="/c/{{ post.community.link() }}" title="{{ post.community.ap_domain }}">{{ post.community.title }}@{{ post.community.ap_domain }}</a></li>
+  </ol>
+</nav>

app/templates/post/_post_full.html

@@ -1,68 +1,7 @@
 <div class="row position-relative post_full">
-    {% if post.type == POST_TYPE_IMAGE -%}
+        <div class="col post_col {% if post.type == POST_TYPE_IMAGE %}post_col post_type_image{% else %}post_type_normal{% endif %}">
-        <div class="col post_col post_type_image">
+            {% include "post/_breadcrumb_nav.html" %}
-            <nav aria-label="breadcrumb" id="breadcrumb_nav" title="Navigation">
-              <ol class="breadcrumb">
-                {% for breadcrumb in breadcrumbs -%}
-                  <li class="breadcrumb-item">{% if breadcrumb.url -%}<a href="{{ breadcrumb.url }}">{% endif -%}{{ breadcrumb.text }}{% if breadcrumb.url -%}</a>{% endif -%}</li>
-                {% endfor -%}
-                <li class="breadcrumb-item"><a href="/c/{{ post.community.link() }}" title="{{ post.community.ap_domain }}">{{ post.community.title }}@{{ post.community.ap_domain }}</a></li>
-                <li class="breadcrumb-item active">{{ post.title|shorten(15) }}</li>
-              </ol>
-            </nav>
-            <h1 class="mt-2 post_title"{% if post.language_id and post.language.code != 'en' %} lang="{{ post.language.code }}"{% endif %}>{{ post.title }}
-                {% if current_user.is_authenticated -%}
-                    {% include 'post/_post_notification_toggle.html' -%}
-                {% endif -%}
-                {% if post.nsfw -%}<span class="warning_badge nsfw" title="{{ _('Not safe for work') }}">nsfw</span>{% endif -%}
-                {% if post.nsfl -%}<span class="warning_badge nsfl" title="{{ _('Potentially emotionally scarring content') }}">nsfl</span>{% endif -%}
-            </h1>
-            {% if post.url -%}
-                <p><a href="{{ post.url }}" rel="nofollow ugc" target="_blank" aria-label="Go to image">{{ post.url|shorten_url }}
-                    <span class="fe fe-external"></span></a></p>
-            {% endif -%}
-            <p>{% if post.reports > 0 and current_user.is_authenticated and post.community.is_moderator(current_user) -%}
-                    <span class="red fe fe-report" title="{{ _('Reported. Check post for issues.') }}"></span>
-                {% endif -%}<small>submitted {{ arrow.get(post.posted_at).humanize(locale=locale) }} by {{ render_username(post.author) }}
-                {% if post.edited_at -%} edited {{ arrow.get(post.edited_at).humanize(locale=locale) }}{% endif -%}
-            </small></p>
-            <div class="post_image">
-                {% if post.image_id -%}
-                    {% if low_bandwidth -%}
-                        <a href="{{ post.image.view_url(resize=True) }}" rel="nofollow ugc"><img src="{{ post.image.medium_url() }}"
-                             alt="{{ post.image.alt_text if post.image.alt_text else post.title }}" fetchpriority="high" referrerpolicy="same-origin"
-                             width="{{ post.image.width }}" height="{{ post.image.height }}" /></a>
-                    {% else -%}
-                        <a href="{{ post.image.view_url() }}" rel="nofollow ugc">
-                            <img src="{{ post.image.view_url(resize=True) }}" lowsrc="{{ post.image.medium_url() }}"
-                                 sizes="(max-width: 512px) 100vw, 854px" srcset="{{ post.image.medium_url() }} 512w, {{ post.image.view_url(resize=True) }} 1024w"
-                                 alt="{{ post.image.alt_text if post.image.alt_text else post.title }}"
-                                 fetchpriority="high" referrerpolicy="same-origin" >
-                        </a>
-                    {% endif -%}
-                {% else -%}
-                    <a href="{{ post.url }}" rel="nofollow ugc" target="_blank" aria-label="Go to image"><img src="{{ post.url }}" style="max-width: 100%; height: auto;"  /></a>
-                {% endif -%}
-            </div>
-            <div class="post_body mt-2"{% if post.language_id and post.language.code != 'en' %} lang="{{ post.language.code }}"{% endif %}>
-                {{ post.body_html|community_links|safe if post.body_html else '' }}
-            </div>
-        </div>
-    {% else -%}
-        <div class="col post_col post_type_normal">
-            <nav aria-label="breadcrumb" id="breadcrumb_nav" title="Navigation">
-              <ol class="breadcrumb">
-                {% for breadcrumb in breadcrumbs -%}
-                  <li class="breadcrumb-item">{% if breadcrumb.url -%}<a href="{{ breadcrumb.url }}">{% endif -%}{{ breadcrumb.text }}{% if breadcrumb.url -%}</a>{% endif -%}</li>
-                {% endfor -%}
-                <li class="breadcrumb-item"><a href="/c/{{ post.community.link() }}">{{ post.community.title }}@{{ post.community.ap_domain }}</a></li>
-                <li class="breadcrumb-item active">{{ post.title|shorten(15) }}</li>
-              </ol>
-            </nav>
             <h1 class="mt-2 post_title" {% if post.language_id and post.language.code != 'en' %}lang="{{ post.language.code }}"{% endif %}>{{ post.title }}
-                {% if current_user.is_authenticated -%}
-                    {% include 'post/_post_notification_toggle.html' -%}
-                {% endif -%}
                 {% if post.nsfw -%}<span class="warning_badge nsfw" title="{{ _('Not safe for work') }}">nsfw</span>{% endif -%}
                 {% if post.nsfl -%}<span class="warning_badge nsfl" title="{{ _('Potentially emotionally scarring content') }}">nsfl</span>{% endif -%}
             </h1>
@@ -71,6 +10,9 @@
                     <a href="{{ post.url }}" target="_blank" rel="nofollow ugc" class="post_link"><img src="{{ post.image.thumbnail_url() }}" alt="{{ post.image.alt_text if post.image.alt_text else '' }}"
                             width="{{ post.image.thumbnail_width }}" height="{{ post.image.thumbnail_height }}" loading="lazy" /></a>
                 </div>
+            {% elif post.type == POST_TYPE_IMAGE and post.url -%}
+                <p><a href="{{ post.url }}" rel="nofollow ugc" target="_blank" aria-label="Go to image">{{ post.url|shorten_url }}
+                    <span class="fe fe-external"></span></a></p>
             {% endif -%}
             <p>{% if post.reports > 0 and current_user.is_authenticated and post.community.is_moderator(current_user) -%}
                     <span class="red fe fe-report" title="{{ _('Reported. Check post for issues.') }}"></span>
@@ -78,9 +20,31 @@
                 {{ render_username(post.author) }}
                 {% if post.edited_at -%} edited {{ arrow.get(post.edited_at).humanize(locale=locale) }}{% endif -%}</small>
             </p>
-            {% if post.type == POST_TYPE_LINK -%}
+            {% if post.type == POST_TYPE_IMAGE -%}
+                <div class="post_image">
+                    {% if post.image_id -%}
+                        {% if low_bandwidth -%}
+                            <a href="{{ post.image.view_url(resize=True) }}" rel="nofollow ugc"><img src="{{ post.image.medium_url() }}"
+                                 alt="{{ post.image.alt_text if post.image.alt_text else post.title }}" fetchpriority="high" referrerpolicy="same-origin"
+                                 width="{{ post.image.width }}" height="{{ post.image.height }}" /></a>
+                        {% else -%}
+                            <a href="{{ post.image.view_url() }}" rel="nofollow ugc">
+                                <img src="{{ post.image.view_url(resize=True) }}" lowsrc="{{ post.image.medium_url() }}"
+                                     sizes="(max-width: 512px) 100vw, 854px" srcset="{{ post.image.medium_url() }} 512w, {{ post.image.view_url(resize=True) }} 1024w"
+                                     alt="{{ post.image.alt_text if post.image.alt_text else post.title }}"
+                                     fetchpriority="high" referrerpolicy="same-origin" >
+                            </a>
+                        {% endif -%}
+                    {% else -%}
+                        <a href="{{ post.url }}" rel="nofollow ugc" target="_blank" aria-label="Go to image"><img src="{{ post.url }}" style="max-width: 100%; height: auto;"  /></a>
+                    {% endif -%}
+                </div>
+            {% elif post.type == POST_TYPE_LINK -%}
                 <p><a href="{{ post.url }}" rel="nofollow ugc" target="_blank" class="post_link" aria-label="Go to post url">{{ post.url|shorten_url }}
-                <span class="fe fe-external"></span></a></p>
+                <span class="fe fe-external"></span></a>
+                    {% if post.domain.post_warning -%}
+                        <span class="fe fe-warning red" title="{{ post.domain.post_warning }}"></span>
+                    {% endif -%}</p>
                 {% if post.url.endswith('.mp3') -%}
                     <p><audio controls preload="{{ 'none' if low_bandwidth else 'metadata' }}" src="{{ post.url }}"></audio></p>
                 {% elif post.url.endswith('.mp4') or post.url.endswith('.webm') -%}
@@ -141,6 +105,9 @@
                 {% if archive_link -%}
                     <p><a href="{{ archive_link }}" rel="nofollow ucg noindex" target="_blank">{{ _('Archive.ph link') }} <span class="fe fe-external"></span></a></p>
                 {% endif -%}
+                {% if post.licence_id -%}
+                    <p>Licence: {{ post.licence.name }}</p>
+                {% endif -%}
             </div>
             {% if post.type == POST_TYPE_POLL -%}
                 <div class="post_poll">
@@ -185,7 +152,6 @@
                 </div>
             {% endif -%}
         </div>
-    {% endif -%}
 
     {% if post.tags.count() > 0 -%}
         <nav role="navigation">
@@ -209,6 +175,13 @@
             <span aria-label="{{ _('Number of cross-posts:') }}">{{ len(post.cross_posts) }}</span></a>
         </div>
         {% endif -%}
-        <a href="{{ url_for('post.post_options', post_id=post.id) }}" class="post_options_link" rel="nofollow"><span class="fe fe-options" title="Options"> </span></a>
+        <div class="notify_toggle">
+            {% if current_user.is_authenticated and current_user.verified -%}
+                {% include 'post/_post_notification_toggle.html' -%}
+            {% endif -%}
+        </div>
+        <div class="post_options_link">
+            <a href="{{ url_for('post.post_options', post_id=post.id) }}" rel="nofollow"><span class="fe fe-options" title="Options"> </span></a>
+        </div>
     </div>
 </div>

app/templates/post/_post_reply_teaser.html

@@ -2,6 +2,8 @@
   teaser: Renders just a teaser
   disable_voting: Disable voting buttons (to prevent mass downvoting)
   no_collapse: Don't collapse for admin and moderator views
+  show_deleted: Show deleted content (for admin views)
+  children: replies to this reply
 #}
 {% if current_user.is_authenticated -%}
     {% set collapsed = ((post_reply.score <= current_user.reply_collapse_threshold) or post_reply.deleted)
@@ -9,7 +11,7 @@
 {% else -%}
     {% set collapsed = (post_reply.score <= -10) and not no_collapse -%}
 {% endif -%}
-<div class="container comment{% if post_reply.score <= -10 %} low_score{% endif %}" id="comment_{{ post_reply.id }}">
+<div class="container comment{% if post_reply.score and post_reply.score <= -10 %} low_score{% endif %}{% if post_reply.author.id == post_reply.post.author.id %} original_poster{% endif %}" id="comment_{{ post_reply.id }}"{% if post_reply.language_id and post_reply.language.code != 'en' %} lang="{{ post_reply.language.code }}"{% endif %} aria-level="{{ post_reply.depth+1 }}" role="treeitem">
     {% if not post_reply.author.indexable -%}<!--googleoff: all-->{% endif -%}
     {% if teaser -%}
     <div class="row">
@@ -25,15 +27,14 @@
     {% endif -%}
     <div class="row">
         <div class="col-auto comment_author">
-            by
+            <span class="visually-hidden">by</span>
             {{ render_username(post_reply.author) }}
             {% if post_reply.author.id == post_reply.post.author.id -%}
                 <span title="Submitter of original post" aria-label="{{ _('Post creator') }}" class="small"> [OP]</span>
             {% endif -%}
         </div>
-        <div class="col-auto text-muted small">
+        <div class="col-auto text-muted small pt-05">
-            {{ arrow.get(post_reply.posted_at).humanize(locale=locale) }}
+            {{ arrow.get(post_reply.posted_at).humanize(locale=locale) }}{% if post_reply.edited_at -%}, edited {{ arrow.get(post_reply.edited_at).humanize(locale=locale) }}{% endif -%}
-            {% if post_reply.edited_at -%}, edited {{ arrow.get(post_reply.edited_at).humanize(locale=locale) }} {% endif -%}
         </div>
         <div class="col-auto">
             {% if post_reply.reports and current_user.is_authenticated and post_reply.post.community.is_moderator(current_user) -%}
@@ -52,17 +53,27 @@
 
     <div class="row comment_body hidable{% if post_reply.reports and current_user.is_authenticated and post_reply.post.community.is_moderator(current_user) %} reported{% endif %}">
         <div class="col-12">
-            {{ post_reply.body_html | community_links | safe }}
+            {% if post_reply.deleted and not show_deleted -%}
+                {% if post_reply.deleted_by is none or post_reply.deleted_by != post_reply.user_id -%}
+                    <p>Deleted by moderator</p>
+                {% else -%}
+                    <p>Deleted by author</p>
+                {% endif -%}
+            {% else -%}
+                {{ post_reply.body_html | community_links | safe }}
+            {% endif -%}
         </div>
     </div>
     <div class="comment_actions hidable">
+        <div class="post_replies_link">
             {% if post_reply.post.comments_enabled -%}
                 {% if not post_reply.post.deleted and not post_reply.deleted -%}
-                    <a href="{{ url_for('post.add_reply', post_id=post_reply.post.id, comment_id=post_reply.id) }}" class="" rel="nofollow noindex"><span class="fe fe-reply"></span> reply</a>
+                    <a href="{{ url_for('post.add_reply', post_id=post_reply.post.id, comment_id=post_reply.id) }}" rel="nofollow noindex"><span class="fe fe-reply"></span> reply</a>
                 {% else -%}
                     <span class="fe fe-reply"></span> reply
                 {% endif -%}
             {% endif -%}
+        </div>
         <div class="voting_buttons_new">
             {% with comment=post_reply, community=post_reply.post.community -%}
                 {% include "post/_comment_voting_buttons.html" -%}
@@ -76,17 +87,36 @@
             {% endif -%}
         </div>
         <div class="notify_toggle">
-            {% if current_user.is_authenticated -%}
+            {% if current_user.is_authenticated and current_user.verified -%}
                 {% with comment=dict(comment=post_reply) -%}
                     {% include "post/_reply_notification_toggle.html" -%}
                 {% endwith -%}
             {% endif -%}
         </div>
-        {% if not post_reply.post.deleted -%}
+        <div class="comment_actions_link">
-            <a href="{{ url_for('post.post_reply_options', post_id=post_reply.post.id, comment_id=post_reply.id) }}" class="comment_actions_link" rel="nofollow noindex" aria-label="{{ _('Comment options') }}"><span class="fe fe-options" title="Options"> </span></a>
+            {% if not post_reply.post.deleted -%}
-        {% endif -%}
+                <a href="{{ url_for('post.post_reply_options', post_id=post_reply.post.id, comment_id=post_reply.id) }}" rel="nofollow noindex" aria-label="{{ _('Comment options') }}"><span class="fe fe-options" title="Options"> </span></a>
+            {% endif -%}
+        </div>
     </div>
     {% if not post_reply.author.indexable -%}<!--googleon all-->{% endif -%}
+
+    {% if children -%}
+        <div class="replies hidable" role="group">
+            {% if not THREAD_CUTOFF_DEPTH or post_reply.depth <= THREAD_CUTOFF_DEPTH -%}
+                    {% for reply in children -%}
+                        {% with post_reply=reply['comment'], children=reply['replies'] %}
+                            {% include 'post/_post_reply_teaser.html' %}
+                        {% endwith %}
+                    {% endfor -%}
+            {% else -%}
+                <div class="continue_thread hidable">
+                    <a href="{{ url_for('post.continue_discussion', post_id=post_reply.post.id, comment_id=post_reply.id, _anchor='replies') }}">Continue thread</a>
+                </div>
+            {% endif -%}
+        </div>
+    {% endif -%}
+
     {% if collapsed -%}
         <script nonce="{{ session['nonce'] }}" type="text/javascript">
             if (typeof(toBeHidden) === 'undefined') {