This new method batch-loads them from the database, and removes the
cache-layer with the intention that a couple of batch fetches from the
DB is more efficient than hitting the cache each time.
Replaced the existing xpath-heavy system with a more manual traversal
approach. Fixes following slow areas of old system:
- Old system would repeat ID-setting action for elements (Headers could
be processed up to three times).
- Old system had a few very open xpath queries for headers.
- Old system would update links on every ID change, which triggers it's
own xpath query for links, leading to exponential scaling issues.
New system only does one xpath query for links when changes are needed.
Added test to cover.
For #3932
Added test to cover.
Did attempt a 64k limit, but values over 2k significantly increase
chance of other issues since this URL may be used in redirect headers.
Would rather catch issues in-app.
For #4044
Altered & updated permissions repo, and existing connected
RoleController to suit.
Also extracts in-app success notifications to auto activity system.
Tweaked tests where required.
Failure of loading drawings will now close the drawing view and show an
error message, hinting at file or permission issues, instead of leaving
the user facing a continuosly loading interface.
Adds test to cover.
This also updates errors from our HTTP service to be wrapped in a custom
error type for better identification and so the error is an actual
javascript error. Should be object compatible.
Related to #3955.
Had misalignment between query and usercan, The nuance between fallback
and entity-role permissions was not taken into account by the query
system. Now added with new test cases to cover.
Is a tweak upon the existing approach, mainly to store and query role
permission access in a way that allows muli-level states that may
override eachother. These states are represented in the new PermissionStatus
class.
This also simplifies how own permissions are stored and queried, to be
part of a single column.
Updated language system to only use initial part of locale for
translation pluralisation to better match the hard-coded logic of the
built-in MessageSelector. Extends and overrides Laravel's default for
this system.
Added test to cover.
Related to #3976.
User roles would only be actioned if they existed in the form request,
hence removal of all roles would have no data to action upon.
This adds a placeholder 0-id role to ensure there is always role data to
send, even when no roles are selected. This field value is latter
filtered out.
Added test to cover.
Likely related to #3922.
Now assume, based on OIDC discovery spec, that keys without 'use' are
'sig' keys. Should not affect existing use-cases since existance of such
keys would have throw exceptions in prev. versions of bookstack.
For #3869
To avoid non-user GET requests (Such as those from email scanners)
auto-triggering the confirm submission. Made auto-submit the form via
JavaScript in this extra added step with user-link backup to keep
existing user flow experience.
Closes#3797
Changes their endpoints and remove the user id from the URLs.
Simplifies list changes to share a single endpoint, which aligns it to
the behaviour of the existing sort preference endpoint.
Also added test to ensure user preferences are deleted on user delete.
Somehow I accidentally deleted previous line 143 in this commit:
3839bf6bf1
which would then break permission generation for content related to, or
containing, chapters in the recycle bin.
Found via user report (subz) & debugging in discord.
- Fixed inheriting control for new system.
- Tested copying shelf permissions to books.
- Added additional handling for inheriting scenario identification.
Also updates show roles on permission view to just those with
permissions applied.
Fixes rounded borders for lone permission rows.
Moves "Everyone Else" handling from role to new class.
Nothing on back-end logic done to hook this new option up.
Addition of permissions for role_id=0 works out of the box, but active
"everyone else" permissions, with no priviliges, is currently not
working. Needs change of permission gen logic also.
Created a generic list formatting helper class for this, to align with
logic used on the search results endpoint and for easier future re-use
in a standardised way.
Also updated some class property types.
Added test to cover new books-contents results.
Related to #3734
- Aligned prefix-type filtering with back-end.
- Increased suggestion search cut-off from 3 to 4.
- Increased amount of suggestions shown.
- Ordered suggestions to be name asc, as you'd expect on search.
- Updated front-end filtering to use full search query, instead of
truncated version, for further front-end filtering capability.
Related to #3720
Removed old pagination provider as url handling now achieved in a better
way.
Removed unused broadcast service provider.
Moved view-based tweaks into specific provider.
Reorganised provider config list.
Brings app auth controller handling aligned within the app, rather than
having many overrides of the framwork packages causing confusion and
messiness over time.
Extracted much of the language and locale work to a seperate, focused class.
Updated php set_locale usage to prioritise UTF8 usage.
Added locale options for windows.
Clarified what's a locale and a bookstack language string.
For #3590 and maybe #3650
Caused by changes to page repo in reference work,
This adds back in the slug generate although at a more central place.
Adds a test case to cover the problematic scenario.
Not totally happy with implementation as is requires extra service to be
injected to core controllers, but does the job.
Included test to cover.
Updated some controller properties to be typed while there.
Updated revision listing to only fetch required fields, massively
reducing memory usage by not loading content.
This also updates user avatar handling to effectively cache the avatar
url within request to avoid re-searching from cache, which may improve
performance of others areas of the application.
This also upates handling of the revisions list view to extract table
row to its own view to break things down a bit.
For #3633
Added additional exceptions to prevent such cases in the future, so
that they are caught in dev ideally.
Added test case specifically for reported favourite scenario.
Is generally aligned with out SAML2 group sync functionality, but for
OIDC based upon feedback in #3004.
Neeeded the tangental addition of being able to define custom scopes on
the initial auth request as some systems use this to provide additional
id token claims such as groups.
Includes tests to cover.
Tested live using Okta.
- Added ability to search books list (Local simple text match).
- Added handles, hover-states and cursor states for better user
interaction and clearer use of drag & drop.
- Improved styles for dark mode.
- Converted shelf sort component to newer component format.
- Modernized shelf controller code a little.
Related to #3266
Both caught in tests:
Fixed loss of permissions for admin users when entity restrictions were
active, since there are no entity-restrictions for the admin role but
we'd force generate them in joint permissions, which would be queried.
Fixed new role permission checks when permissions given with only the
action (eg. 'view'), since the type prefix would be required for role
permission checks. Was previously not needed as only the simpler form
was used in the jointpermissions after merge & calculation.
Cleaned up PermissionApplicator to remove old cache system which was
hardly ever actuall caching anything since it was reset after each
public method run.
Changed the scope of 'userCanOnAny' to just check entity permissions,
and added protections of action scope creep, in case a role permission
action was passed by mistake.
Updated ajax search and entity selector usage to display and handle
items that the user does not have permission to interact with.
Started logic changes to not allow permission type to be passed around,
with views instead being the fixed sole permission.
For PR #3406
- Updated naming from 'redirect' to 'initate/initation'.
- Updated phpunit.xml and .env.example.complete files with the new
option.
- Cleaned up controller logic a bit.
- Added content and design to the new initation view to not leave user
on a blank view for a while.
- Added non-JS button to initiation view as fallback option for
progression.
- Moved new test to it's own Test class and expanded with additional
scenario tests for better functionality coverage.
Linebreaks would previously essentially be removed during index and
hence joined to adjacent words, breaking prefix matching.
Added test to cover.
For #3508
- Also extracted shelf to book view elements to own partial.
- Fixed some existing logic including image param handling in update
request and activity logging against correct element.
- Updates book/shelf cover image handling for easier cloning/handling.
- Adds core logic for promoting books/chapters up a level.
- Enables usage of book/shelf cover image via API.
Related to #1087
Moved export templates elements into their own folder for better
grouping of logical usage.
Within the base export template, added some body classes to allow easier
targeted customisation via custom head css.
Split content of export templates into smaller partials for easier
future customization.
Closes#3443
Was reporting wrong due to use of old polymorphic namespace references.
Test was not picking up as assertElementContains had wider scope than
expected, looking within the HTML of the element instead of the text
which you might expect. Updated test helper to look at text instead.
Updated default value for APP_URL so that the startsWith call is not
passed null, since that causes deprecation notice in PHP8.1.
Would show when APP_URL was not set, adding extra confusiion.
Dumping details that were binary, such as the jpegphoto data, would
cause the dump to fail on the encoding to JSON.
This change forces content to be UTF8 before dumping.
Updated existing test to cover.
Closes#3396
Call to `ob_end_clean` would error if the environment did not use the
PHP `output_buffering` option. This adds an additional check and updates
the comment to be more specific to the exact scenario of the condition.
Tested with output_buffering=Off and output_buffering=4096
Closes#3415
Made the following changes, many of these are just to align with
existing conventions.
- Updated urls to be hypenated, instead of underscored, to match other system endpoints.
- Updated URL parameter to be `deletionId` instead of `id`, and removed the ID-based comment on controller methods, so the required ID model is clear from the URL alone, since its not clear from the URL endpoint alone like existing endpoints. This follows the pattern used in the "web" routes.
- Added extra detail on some controller method comments, and copied permission comment to each method.
- Removed existing field visibility mechanisms to use simpler model-based visibility since we didn't need anything too special here (After some of my other changes).
- Allowed the "deletable" model to be shown in response to provide a little more detail on the main deleted item.
- Updated parent/child-count loading to be on the "deletable" model instead of additional properties which results in simpler controller logic and enforces the idea these are relations on the deletable, not the deletion itself. It also removes additional exposure of model namespacing.
- Updated (int) casts to intval, just since that's our most common conversion method in the codebase.
- Testing: Removed `actingAsAuthorizedUser` and used the admin user instead to prevent extra auth steps on each test.
- Testing: Cut logic/data-checks from tests if already covered by other tests.
- Testing: Added simple assertions for delete/restore response data.
- Examples: Updated list example to reflect changes.
Review of PR #3377
To be followed up with changes to polymorphic relations to hide
namespacing.
Extracted page editor view data gathering to its own class for
alignment. Updated the data used in views as part of the process to use
view-specific variables instead of custom attributes added to models.
Also moved tinymce library loading so it's not loaded when not using the
wysiwyg editor.
Decided it's relevant to entity updated_at since tags are now indexed
alongside content.
- Also fixed tags not applied on shelf.
- Also enforced proper page API update validation.
- Adds tests to cover.
For #3319Fixes#3370
- Added testing check to buffer stop/clear on streaming output due to
interference during tests.
- Made content-disposition header a little safer in download responses.
- Also aligned how we check for testing environment.
Fixes hitting memory limits where downloaded file sizes are much greater
than memory limit. Stopping and flushing output buffer seemed to stop
limits causing issues when fpassthru is used.
Tested with 24M memory limit and 734M file
This allows download of attachments that are larger than current memory
limits, since we're not loading the entire file into memory any more.
For inline file responses, we take a 1kb portion of the file to sniff
before to check mime before we proceed.
This updates the custom Request handler to provide only the scheme and
host on the `getSchemeAndHttpHost` call, instead of providing the whole
APP_URL value, while adding an override to the 'getBaseUrl' to use the
APP_URL content instead of the guessed/detected Symfony value.
Untested apart from simple local setup.
Related to #2765
- Fixed issue where redirect for `/settings` view would not be ran
through base url generator so would not create a correct path in some
cases. Now routed through controller with normal redirect.
- Fixed custom head content being active on settings pages due to route
name changes, for when viewing settings, in last release.
Fixes#3356 and #3355
- Updated PhpStan PHP version option to match project.
- Applied StyleCI changes.
- Updated static to self in WebhookFormatter, following static analysis
guidance.
- Fixed mis-matched header tags.
- Split settings out to new views using a core shared layout.
- Extracted added language text to translation files.
- Updated settings routes to be dynamic to category.
- Added redirect for old primary settings route.
- Updated existing tests to cover settings route changes.
- Added tests to cover settings view.
- Improved contrast of settings links for dark mode.
Creates a new organsied formatting system for webhook data, with
interfaces for extending with custom model formatting rules.
Allows easy usage & extension of the default bookstack formatting
behaviour when customizing webhook events via theme system, and keeps
default data customizations organised.
This also makes the following webhook data changes:
- owned_by/created_by/updated_by user details are loaded for events with
Entity details. (POTENTIALLY BREAKING CHANGE).
- current_revision details are loaded for page update/create events.
Added testing to cover added model formatting rules.
For #3279 and #3218
Uses original image data to extract orientation exif to apply image
transformations before scaling and save. Manually done due to issues
with exif data loss during the existing Invervention image path.
For #1854
- Configurable via 'ALLOWED_IFRAME_SOURCES' .env option.
- Also updated how CSP rules are set, with a single header being used
instead of many.
- Also applied CSP rules to HTML export outputs.
- Updated tests to cover.
For #3314
- For some reason, TinyMCE would handle empty paragraphs with a ' '
by default but this would be removed when the paragraph had an
attribute. This was fine in the old editor.
- This changes the approach to use '<br>' tags within elements
for "spaced emptiness".
- For compatbility with any existing empty paragraphs, I updated the
styles to show default height for empty paragraph sections.
- This also makes changes to help preserve encoded html tags
since they were getting converted along the journey.
Related to #3302
Fixes issue where certain errors would not show to the user
due to extra navigation jumps which lost the error message
in the process.
This simplifies and aligns exceptions with more directly
handled exception usage at the controller level.
Fixes#3264
- Removed old 'editor-*-update' commands to instead use the aligned
'editor::replace' command that we already have.
- Changed the way custom styles are loaded for the WYSIWYG editor so we
don't need an API call but instead scape content from the parent page
header using comments as identifiers. Added tests to ensure comments
exist and align.
- Required extracting logic into repo.
- Changed some existing creation paths to standardise behaviour.
- Added test to cover new endpoint.
- Added extra test for user delete to test migration.
- Changed how permission errors are thrown to ensure the right status
code can be reported when handled in API.
- Required changing the docs generator to handle more complex
object-style rules. Bit of a hack for some types (password).
- Extracted core update logic to repo for sharing with API.
- Moved user update language string to align with activity/logging
system.
- Added tests to cover.
- Updated routes to use new format.
- Changed how hidden fields are exposed to be more flexible to different
use-cases.
- Updated properties available on read/list results.
- Started adding testing coverage.
- Removed old unused UserRepo 'getAllUsers' function.
Related to #2701, Progression of #2734
- Added tests to cover.
- Refactored some existing testing.
- Requires password or external_auth_id to be provided. Defaults to
password.
- Randomly sets password to 32 digit random chars if external_auth_id
provided instead.
For #3222
Further fixes for #3120, Adds DOMPDF specific adjustments to prevent
full width linked images being cut-off as per last tweak.
This does not fix usage in smaller cases (tables) but tested on
master DOMPDF branch shows that will likely be fixed in next DOMPDF
upstream release.
DOMPDF fixes would break WKHTMLTOPDF presentation so system updated
to conditionally apply styles.
Updated the getUrl method on deletions to not require any passed
params to align with usage in webhooks.
Probably better to have a proper interface but would require a wider
change.
Fixes#3154
Custom homepage usage will now be checked before any actioning
of deletion rather than potentially causing an exception acting
during the deletion.
Previously a deletion could still be created, within the recycle bin,
for the parent which may lead to the page being deleted anyway.
For #3150
- Aligned permissions control with move operations to check
delete/create permissions against old/new locations.
- Added tests to cover additional permissions scenarios.
Previous implemenations were hard to read so changing to be more
logically simplistic. Still needs further coverage in tests and
review/alignment of permissions to use.
Models are now loaded into their own map to then be used for sorting and
reporting back of changed books. Prevents akward logic ordering issues
of before where some bits of code assumed/hoped for loaded models on
abstract data structures.
New levels of permissions are now checked for items within the
sort operation. Needs testing to cover.
Adds apng sniffing when generating thumbnails with retained ratios to
serve the original image files, as we do for GIF images, to prevent
the image being resized to a static version.
Is more tricky than GIF since apng file mimes and extensions
are the same as png, we have to detect part of the file header
to sniff the type. Means we have to sniff at a later stage
than GIF since we have to load the image file data.
Made some changes to the image thubmnail caching while doing
this work to fit in with this handling.
Added test to cover.
For #3136.
- Added a user-configurable timeout option to webhooks.
- Added webhook fields for last-call/error datetime, in addition to last
error string, which are shown on webhook edit view.
Related to #3122
- Copies via loading in model on create view.
- Updated role views while editing to bring up to similar format as
that used for more modern app areas.
- Added tests to cover.
Related to #1123
Updated all password validation to use central password defaults
system while updating length requirements to now all match
at 8 characters minimum.
Some language text was technically correct (More than 7 characters)
but this has been updated for clarity and to prompt other translations
to be updated.
Closes#2237
Altered access & usage of the /search/users/select endpoint with the
following changes:
- Removed searching of email address to prevent email detail discovery
via hunting via search queries.
- Required the user to be logged in and have permission to manage users
or manage permissions on items in some way.
- Removed the user migration option on user delete unless they have
permission to manage users.
For #3108
Reported in https://huntr.dev/bounties/135f2d7d-ab0b-4351-99b9-889efac46fca/
Reported by @haxatron
- Renamed to "ActivityLogger" to be more focused in usage.
- Extracted out query elements to seperate "ActivityQueries" class.
- Removed old 'addForEntity' activity method to limit activity record
points.
Allows easy direct linking and usage of the HTML preview content
we show in the UI when viewing search results.
Note: preview_html content is a rough representation only, it does not
match exactly what was matched in the database-search-operation which
finds the results.
For #3096 and #3080
Page-related items added on drafts could be visible in certain scenarios
since the applied permissions query filters would not consider
page draft visibility.
This commit alters queries on related items to apply such filtering.
Included test to cover API scenario.
Thanks to @haxatron for reporting.
- Replaced iframe elements with anchor elements wrapped in a paragraph.
- Extracted PDF generation action to seperate class for easier mocking
within testing.
- Added test to cover.
For #3077
Old command registration method was interfering with default commands,
causing only a limited subset of commands to show overall.
This change follows the method the frameworks uses when loading in from a
directory to prevent issues with run/load order.
Changed how the command registration was handled due to complications of
action order found during testing. Now the theme service will resolve
and directly register the command on the Kernel instead of them being
fetched from the ThemeService from within Kernel.
More direct, Seems to work.
Updated the back button to be a proper link instead of a reference to
the last viewed URL since it could break if the last page was the
current one (On validation for example).
Includes test to cover.
Also applied some styleCI changes.
Fixes#2834
Was done for convenience but could potentially be exploited by an
attacker using signing up via one of these routes, then forwarding
an email confirmation to another user so they unknowingly utilise
an account someone else controls.
Tweaks the flow of confirming email, and the user invite flow.
For #3050
This change means that code blocks are now included still wrapped in
their pre tags, as we do for tables and lists.
Previously the <code> inner content would be included which would lead
to a generally bad/broken presentation.
Hopefully should not be a breaking change as section include tags for
code was tricky to get to, and it was in a semi-broken state.
For #2406
Replaces the old suggestion of setting JS head 'window.uploadLimit'
variable. This new env option will be used by back-end validation and
front-end libs/logic too.
Limits already likely exist within prod environments at a PHP and
webserver level but this allows an app-level limit and centralises the
option on the BookStack side into the .env
Closes#3033
As per #3047.
Also made some SAML specific fixes:
- IDP initiated login was broken due to forced default session value.
Double checked against OneLogin lib docs that this reverted logic was fine.
- Changed how the saml login flow works to use 'withoutMiddleware' on
the route instead of hacking out the session driver. This was due to
the array driver (previously used for the hack) no longer being
considered non-persistent.
Is a little awkward, emulates a 'list' API endpoint but has unstable
paging and does not support filters/sort. This is detailed on the
endpoint though.
Made some updates to the docs system to better support parameters
and examples on GET requests.
Includes tests to cover.
For #909
Removing prefix route groups out of visual preference.
Those don't really save much and I prefer seeing the complete
paths when going down the list to better guage where I am.
Markdown content was not being stored, only the sent
HTML representation, causing the draft to show as HTML upon next edit.
Added test to cover.
Fixes#3054
- Updated page names to not be limited to a certain length.
- Added better start/end fill logic.
- Prevented <strong> tags from being counted towards the target content
length desired from the formatter.
Created a new class to manage formatting of content for search results.
Turned out to be quite a complex task. This only does the preview text
so far, not titles or tags.
Not yet tested.
This adds parsing of page content so that headers apply a boost to
scores in the search term index.
Additionally, this merges title and content terms to reduce the amount
of stored terms a little.
Includes testing to cover.
Checked over latest changes for potential SQL injection, all variable
usages are either (from trusted sourced AND case) or using
parameters/bindings to ensure it's handled at driver/lib level.
Relations now loaded during back-end query phase instead of being lazy
loaded one-by-one within views.
Reduced queries in testing from ~60 to ~20.
Need to check other areas list-item.php's "showPath" option is used to
ensure relations are properly loaded for those listings.
Adds adjustment of search term 'score' (Using in result ranking) so that
a relative 0.3 to 1.3 mulitplier is applied based upon relative
popularity within the whole database. At this point the term popularity
is still done via a prefix match against the search term.
Uses a SUM(IF(cond, a, IF(cond, a, ...))) chain to produce the scoring
result in the select query.
