Commit Graph

229 Commits

Author SHA1 Message Date
Chlins Zhang
43ccd2f09f
perf: optimize the performance of accessory query (#19557)
1. Add db index for subject_artifact_id in table artifact_accessory.
2. Optimize the SQL of excluding the accessory for artifact count.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-11-22 02:11:11 +00:00
Wang Yan
ed370a496b
deprecate resource label (#19349)
There is no api is using the DAO, remove it from the source code.

Signed-off-by: wang yan <wangyan@vmware.com>
2023-09-14 01:25:52 +08:00
stonezdj(Daojun Zhang)
0e92eaea18
Sort most dangerous vulnerabilities by score and severity level (#19103)
Check it is referred by a scan_report
  Add index to report_vulnerability_record to improve performance
  Fixes #19014 #19028

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-08-07 10:13:34 +08:00
stonezdj(Daojun Zhang)
c25a7ca68d
Skip to run migrate script when data available (#18976) 2023-07-25 19:41:35 +08:00
stonezdj(Daojun Zhang)
5e4163b53b
Create index in vulnerability_record table (#18949)
add index for better query performance

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-07-20 14:00:40 +08:00
Chlins Zhang
970bdab936
fix: correct the operator in the webhook payload (#18906)
Fix the incorrect or meaningless operator in the webhook payload.

Fixes: #18438

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-07-19 15:40:29 +08:00
stonezdj(Daojun Zhang)
d4aa9b13c4
Add vulnerability search API (#18924)
use q.Query to pass all query conditions

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-07-19 10:17:14 +08:00
stonezdj(Daojun Zhang)
d84b1d07d2
Update table scan_report and extract cvss_v3_score from vendor attribute (#18854)
For better performance when query cve information, add summary information to scan_report
    Extract cve_score from vendor attribute in vulnerability_record
    SQL migrate script for the update

Signed-off-by: stonezdj <daojunz@vmware.com>
2023-06-29 17:30:50 +08:00
Chlins Zhang
2958d7b7ff
fix: cherry pick the migration sql (#18644)
Cherry pick the migration SQL of 2.8.1 to main branch.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-05-10 16:35:07 +08:00
Chlins Zhang
36c2b93fe6
fix: sweep executions of image scan job (#18649)
1. Change the SCAN_ALL job execution retain counts from 5 to 1(per
   current design, only one report be stored for every artifact, so
   retain latest 1 is enough).
2. Enable the sweep for IMAGE_SCAN job(retain latest 1).

Fixes: #18633

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-05-10 13:23:54 +08:00
Chlins Zhang
362387f914
fix: improve the performance of list artifacts (#18610)
1. Change the query for listing tasks of scan which can use the db
   index.
2. Add the gin index for task.extra_attrs.report_uuids

Fixes: #18013

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-04-30 09:10:28 +08:00
Chlins Zhang
72a6ef2514
style: adjust the style of migration SQL (#18475)
Signed-off-by: chlins <chenyuzh@vmware.com>
2023-04-04 17:37:27 +08:00
Chlins Zhang
cb0749c7ab
fix: add default payload_format for http type webhook (#18445)
1. Add migration SQL to handle the lost payload format for old policies.
2. Set payload format to 'Default' if not specified for http webhook in the API handler.
3. Fix the migration sql of notification_job

Fixes: #18401, #18453

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-04-02 15:33:33 +08:00
Wang Yan
deaecf2de5
add subject artifact repo (#18394)
add suject_artifact_repo column in the table artifact_accessory

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-22 21:48:09 +08:00
Wang Yan
c1d297b015
fix 2.8 migration issue (#18389)
The sql must be idempotent

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-22 16:12:36 +08:00
Wang Yan
2f380495bf
revert subject id in the accessory (#18377)
DO not replact id with digest and just add digest into the accessory table

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-22 10:58:30 +08:00
Chlins Zhang
ff01efc777
fix: fix the webhook migration sql (#18374)
Signed-off-by: chlins <chenyuzh@vmware.com>
2023-03-19 10:27:06 +08:00
Chlins Zhang
2148ee9fbe
migration: add sql to migrate the old webhook job to new way (#18342)
Signed-off-by: chlins <chenyuzh@vmware.com>
2023-03-16 16:27:43 +08:00
Yang Jiao
1e38565aae
Migrate robot account and notification policy related to Chartmuseum (#18316)
Migrate robot permission and notification policy related to Chartmuseum

Update robot permission and notification policy as Harbor v2.8.0 deprecates chartmuseum.
Harbor deprecates chartmuseum as of v2.8.0
Epic: goharbor#17958
Discussion: goharbor#15057

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2023-03-10 12:19:11 +08:00
Wang Yan
295260b7a3
replace subject id with digest (#18278)
Since it has to support push subject and accessories in either order, it has to replace digest with id

Signed-off-by: Wang Yan <wangyan@vmware.com>
2023-03-08 17:03:12 +08:00
Yang Jiao
1f3f732bd6
Update the replication rule related to the Chartmuseum (#18274)
Update the registry and replication rule related to the Chartmuseum

Update replication_policy and registry as Harbor v2.8.0 deprecates chartmuseum.
Harbor deprecates chartmuseum as of v2.8.0
Epic: #17958
Discussion: #15057

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
2023-03-02 17:28:27 +08:00
Wang Yan
74dfc905ec
remove the redundant data from table artifact_blob (#17937)
use sql in the migration process to delete all the useless data of table artifact_blob

Signed-off-by: Wang Yan <wangyan@vmware.com>

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-12-13 15:32:57 +08:00
stonezdj(Daojun Zhang)
57aff88eb5
Add REST API to list job queue status, pause/resume job queue and list schedulers (#17707)
Add queue manager and redis client
  Update scheduler to add count and list

Signed-off-by: stonezdj <daojunz@vmware.com>

Signed-off-by: stonezdj <daojunz@vmware.com>
2022-11-17 20:12:07 +08:00
Chlins Zhang
e81067badb
fix: cherry pick 262 sql for migrations (#17735)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-03 10:05:46 +08:00
Chlins Zhang
c330b8c63a
feat: introduce the copy by chunk for replication (#17602)
1. Add sql migration to alter replication policy table
2. Implement the PullBlobChunk and PushBlobChunk for the underlying v2 registry client
3. Update image transfer logic to support copy by chunk
4. Update the replication policy API handler

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-11-01 11:19:17 +08:00
Chenyu Zhang
70a95a9696
fix: reduce the high db cpu usage for tag retention (#17296)
1. Add two indexes to database migrations.
2. Skip refresh quota in middleware for requests from jobservice.
3. Refresh quota by self in the end of tag retention job.

Closes: #14708

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-04 10:08:02 +08:00
Chenyu Zhang
41d10d571c
fix: repair execution status when it inconsistent (#17128)
Add migrations sql to repair the execution status when it does not
consistent with task status.

Closes: #17114

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-06 17:02:03 +08:00
stonezdj(Daojun Zhang)
8115484f32
Create index for performance (#17022)
Add index to avoid full table scan
Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-04 14:38:05 +08:00
Chenyu Zhang
0cf036e73a
migrate: add db index on artifact repository name (#17053)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-22 14:34:40 +08:00
Chenyu Zhang
5de1a62002
migrate: add db index on artifact repository name (#17035)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-20 16:01:04 +08:00
prahaladdarkin
27ec871185
System Artifact Manager database schema creation, model definitons, and tests (#16678)
Closes:
https://github.com/goharbor/harbor/issues/16540
https://github.com/goharbor/harbor/issues/16541
https://github.com/goharbor/harbor/issues/16542

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-05-09 17:32:57 +08:00
chlins
f7d10474d1 migrations: correct project metadata public value
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-03-28 15:17:20 +08:00
stonezdj
9d480a84d2 Create index on task with job_id
Missing index with job_id, when query task with job_id, it cause a full table scan, caused performance issue
  Fixes #15271

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-03-02 20:51:36 +08:00
Shengwen Yu
aacebcc8c4 fix: vulnerability_record & report_vulnerability_record table "integer out of range" error
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-02-09 10:19:46 +08:00
He Weiwei
8554b7d7c5
refactor: replace lib/pq with jackc/pgx (#16267)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-01-27 11:09:37 +08:00
He Weiwei
89fe7b0819
Move removing old version scan reports of trivy to 2.4.2 (#16261)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-01-24 14:05:24 +08:00
He Weiwei
097efb201b
fix: Clean old version scan reports of trivy (#16235)
From harbor 2.4, the trivy does not support scan report v1.0, we need to remove the old version scan reports from the DB.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-01-17 13:59:28 +08:00
Wang Yan
742e7ded00
add accessory dao service (#16045)
Signed-off-by: wang yan <wangyan@vmware.com>
2021-12-03 14:34:02 +08:00
He Weiwei
b390112f5a
fix: convert severity from negligible to none before saving to db (#15791)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-10-14 16:02:38 +08:00
stonezdj
06715af303 Allow empty email attribute for ldap/oidc user
Define user.Email as sql.NullString to avoid unique constraint when email is empty in LDAP/OIDC
  Separate the common/models/User with the pkg/user/dao/User
  Fixes #10400
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-09-30 14:35:18 +08:00
He Weiwei
ecc1a04c92
refactor: implement a lock free quota (#15399)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-09-17 11:26:37 +08:00
Ziming Zhang
98cef43ead limit replication bandwidth
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-09-14 11:23:28 +08:00
stonezdj
c163bc8317 Delete users under auth_mode other than db_auth
The following information should cleanup before delete user:
  Delete project member of this user.
  Delete oidc_user when auth_mode is oidc_auth.
  Fixes #8424
  It also removes the deleted user from project member and the deleted condition in the project member query for consistency
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-07-12 11:31:40 +08:00
ChenYu Zhang
a038ba672f
perf: add 2.2.3 schema up sql to master (#15027)
Signed-off-by: chlins <chlins.zhang@gmail.com>
2021-06-01 14:12:25 +08:00
Wenkai Yin
c585f92034 Fix bug of migration sql for replication policy
Fix bug of migration sql for replication policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-26 18:57:23 +08:00
Wang Yan
73bd373a75
create index for audit (#14930)
Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-05-20 19:50:13 +08:00
Wenkai Yin
dc059a9a8f Improve the performance of artifact related APIs
Improve the performance of artifact related APIs by adding indexes and refactoring sql logic

Closes #13890 #14813 #14814

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-20 11:25:43 +08:00
Wenkai Yin
c04f3a2aac Fix duplicate execution record issue
When the core service cannot response the checkin request in time, duplicated execution records may be created, this commit introduces the revision column to make sure there is only one record for one schedule trigger

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-12 15:40:52 +08:00
He Weiwei
476732df89 fix: change art_size to bigint in migrations
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-05-11 04:01:02 +00:00
Wenkai Yin
710c80078b Refactor the replication policy destination namespace logic
Support specifying what part of the repository will be replaced by the provided namespace

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-04-28 10:16:20 +08:00