Introduce the quota update provider, improve the performance of pushing
artifacts to same project with high concurrency by implementing
optimistic lock in redis. By default the function is disabled, open it
by set env 'QUOTA_UPDATE_PROVIDER=Redis' for the core container.
Fixes: #18440
Signed-off-by: chlins <chenyuzh@vmware.com>
The format of ScannerRegistration.properties.url should be
`uri` but not `url`.
Fixes: #18798
Signed-off-by: bin liu <liubin0329@gmail.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
fixes#18865
the response header OCI-Subject to indicate to the client that the registry processed the request's subject.
Signed-off-by: wang yan <wangyan@vmware.com>
Co-authored-by: System Administrator <root@wangyanCQFQY.vmware.com>
For better performance when query cve information, add summary information to scan_report
Extract cve_score from vendor attribute in vulnerability_record
SQL migrate script for the update
Signed-off-by: stonezdj <daojunz@vmware.com>
jobservice: add DB to job logger config in readme
Job logger support file, db and stdout, the comment should
include the type of DB as well as file and stdout.
Signed-off-by: bin liu <liubin0329@gmail.com>
1. Related #18779
2. Show how many blobs and manifest have been deleted and how much space has been freed up
Signed-off-by: AllForNothing <sshijun@vmware.com>
Fix the scanAll cannot be stopped in case of large number of artifacts,
add the checkpoint before submit scan tasks, mark the scanAll stopped
flag in the redis.
Fixes: #18044
Signed-off-by: chlins <chenyuzh@vmware.com>
Show more infors in the gc history, like the sweep size and how many blobs and manifests were removed by GC.
Signed-off-by: Wang Yan <wangyan@vmware.com>
Cleanup the associated resources(scan executions and scan reports) after
deletion of artifact.
Fixes: #18634
Signed-off-by: chlins <chenyuzh@vmware.com>
Since harbor deprecates notary since v2.9.0, this pull request targets to remove the code related with notary.
Signed-off-by: Wang Yan <wangyan@vmware.com>
At the time of making this commit, the package `github.com/ghodss/yaml`
is no longer actively maintained.
`sigs.k8s.io/yaml` is a permanent fork of `ghodss/yaml` and is actively
maintained by Kubernetes SIG.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
Co-authored-by: MinerYang <yminer@vmware.com>
1. Change the SCAN_ALL job execution retain counts from 5 to 1(per
current design, only one report be stored for every artifact, so
retain latest 1 is enough).
2. Enable the sweep for IMAGE_SCAN job(retain latest 1).
Fixes: #18633
Signed-off-by: chlins <chenyuzh@vmware.com>
1. Change the query for listing tasks of scan which can use the db
index.
2. Add the gin index for task.extra_attrs.report_uuids
Fixes: #18013
Signed-off-by: chlins <chenyuzh@vmware.com>
feat: log trace ID
Implements #18029
If the Trace ID is sent to the Harbor in HTTP header or the tracing
is enabled (and the Trace ID is generated), the Trace ID will be
added to the log lines as a new field.
Signed-off-by: Peter Gillich <pgillich@gmail.com>
Update the stopped execution status immediately becasue the user
experience is not good if wait long time when stop or reschedule a job.
Fixes: #18526
Signed-off-by: chlins <chenyuzh@vmware.com>
Change the bitSize from 16 to 64 in the jobservice reaper, the 16 is too
small when the redis cursor over the max value of int16.
Fixes: #18486
Signed-off-by: chlins <chenyuzh@vmware.com>
Refactor the logic for updating the status of execution when receiving
the hook from jobservice, avoid the optimistic lock due to the multiple
tasks update one execution by refreshing the status asynchronously. But
still retain the old way by specifying the flag from ENV.
Fixes: #17584
Signed-off-by: chlins <chenyuzh@vmware.com>
Wrap orm tx when the scheduler try to create the task because submit job
maybe failure depends on the jobservice.
Fixes: #18452
Signed-off-by: chlins <chenyuzh@vmware.com>
Fix the incorrect number of total and retained in the retention webhook
payload, and completes the deleted_artifacts field.
Fixes: #18428
Signed-off-by: chlins <chenyuzh@vmware.com>
1. extend the width for event type label
2. add quotes for Retain image last pull time to make it more understandable
Signed-off-by: AllForNothing <sshijun@vmware.com>
1. Change the the value of CloudEvents id from requestid to uuid
2. Add additional requestid to trace to event
Fixes: #18448
Signed-off-by: chlins <chenyuzh@vmware.com>
Co-authored-by: Wang Yan <wangyan@vmware.com>
To enable the middleware to save the project_blob data, make sure to set the accessories options to true when handling the artifact copy.
Signed-off-by: Wang Yan <wangyan@vmware.com>
1. Add migration SQL to handle the lost payload format for old policies.
2. Set payload format to 'Default' if not specified for http webhook in the API handler.
3. Fix the migration sql of notification_job
Fixes: #18401, #18453
Signed-off-by: chlins <chenyuzh@vmware.com>
1. Show a app level warning if there is a stuck job
2. Chang `Replication finished` to `Replication status changed`
Signed-off-by: AllForNothing <sshijun@vmware.com>
Change the JobId param type from int to string, use int will bring some
problems for encode/decode type mismatch which generate the invalid
repository name.
Fixes: #18380
Signed-off-by: chlins <chenyuzh@vmware.com>
1, add fitler artifactType to header when the api is called with filter
2, give an empty json body on non aritfact scenario
3, give an empty array on non accessory scenario
4, fix the artifact type filter issue
Signed-off-by: Wang Yan <wangyan@vmware.com>
refactor: refactor the old goroutine execution sweep to global execution sweep job
1. Delete the old goroutine execution sweeper when create execution.(in the case of high concurrency can cause goroutine backlogs, affect the performance of core)
2. Introduce the new way to sweep executions, a global scheduled job will take the work.
Signed-off-by: chlins <chenyuzh@vmware.com>
As for the distribution spec 1.1, it supports client to push an manifest with subject field. By leverging this fidle, harbor could build up the linkage between the subject artifact and it's accessories.
Signed-off-by: wang yan <wangyan@vmware.com>
