Commit Graph

204 Commits

Author SHA1 Message Date
Chenyu Zhang
70a95a9696
fix: reduce the high db cpu usage for tag retention (#17296)
1. Add two indexes to database migrations.
2. Skip refresh quota in middleware for requests from jobservice.
3. Refresh quota by self in the end of tag retention job.

Closes: #14708

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-08-04 10:08:02 +08:00
Chenyu Zhang
41d10d571c
fix: repair execution status when it inconsistent (#17128)
Add migrations sql to repair the execution status when it does not
consistent with task status.

Closes: #17114

Signed-off-by: chlins <chenyuzh@vmware.com>
2022-07-06 17:02:03 +08:00
stonezdj(Daojun Zhang)
8115484f32
Create index for performance (#17022)
Add index to avoid full table scan
Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-07-04 14:38:05 +08:00
Chenyu Zhang
0cf036e73a
migrate: add db index on artifact repository name (#17053)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-22 14:34:40 +08:00
Chenyu Zhang
5de1a62002
migrate: add db index on artifact repository name (#17035)
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-06-20 16:01:04 +08:00
prahaladdarkin
27ec871185
System Artifact Manager database schema creation, model definitons, and tests (#16678)
Closes:
https://github.com/goharbor/harbor/issues/16540
https://github.com/goharbor/harbor/issues/16541
https://github.com/goharbor/harbor/issues/16542

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2022-05-09 17:32:57 +08:00
chlins
f7d10474d1 migrations: correct project metadata public value
Signed-off-by: chlins <chenyuzh@vmware.com>
2022-03-28 15:17:20 +08:00
stonezdj
9d480a84d2 Create index on task with job_id
Missing index with job_id, when query task with job_id, it cause a full table scan, caused performance issue
  Fixes #15271

Signed-off-by: stonezdj <stonezdj@gmail.com>
2022-03-02 20:51:36 +08:00
Shengwen Yu
aacebcc8c4 fix: vulnerability_record & report_vulnerability_record table "integer out of range" error
Signed-off-by: Shengwen Yu <yshengwen@vmware.com>
2022-02-09 10:19:46 +08:00
He Weiwei
8554b7d7c5
refactor: replace lib/pq with jackc/pgx (#16267)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-01-27 11:09:37 +08:00
He Weiwei
89fe7b0819
Move removing old version scan reports of trivy to 2.4.2 (#16261)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-01-24 14:05:24 +08:00
He Weiwei
097efb201b
fix: Clean old version scan reports of trivy (#16235)
From harbor 2.4, the trivy does not support scan report v1.0, we need to remove the old version scan reports from the DB.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2022-01-17 13:59:28 +08:00
Wang Yan
742e7ded00
add accessory dao service (#16045)
Signed-off-by: wang yan <wangyan@vmware.com>
2021-12-03 14:34:02 +08:00
He Weiwei
b390112f5a
fix: convert severity from negligible to none before saving to db (#15791)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-10-14 16:02:38 +08:00
stonezdj
06715af303 Allow empty email attribute for ldap/oidc user
Define user.Email as sql.NullString to avoid unique constraint when email is empty in LDAP/OIDC
  Separate the common/models/User with the pkg/user/dao/User
  Fixes #10400
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-09-30 14:35:18 +08:00
He Weiwei
ecc1a04c92
refactor: implement a lock free quota (#15399)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-09-17 11:26:37 +08:00
Ziming Zhang
98cef43ead limit replication bandwidth
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-09-14 11:23:28 +08:00
stonezdj
c163bc8317 Delete users under auth_mode other than db_auth
The following information should cleanup before delete user:
  Delete project member of this user.
  Delete oidc_user when auth_mode is oidc_auth.
  Fixes #8424
  It also removes the deleted user from project member and the deleted condition in the project member query for consistency
Signed-off-by: stonezdj <stonezdj@gmail.com>
2021-07-12 11:31:40 +08:00
ChenYu Zhang
a038ba672f
perf: add 2.2.3 schema up sql to master (#15027)
Signed-off-by: chlins <chlins.zhang@gmail.com>
2021-06-01 14:12:25 +08:00
Wenkai Yin
c585f92034 Fix bug of migration sql for replication policy
Fix bug of migration sql for replication policy

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-26 18:57:23 +08:00
Wang Yan
73bd373a75
create index for audit (#14930)
Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-05-20 19:50:13 +08:00
Wenkai Yin
dc059a9a8f Improve the performance of artifact related APIs
Improve the performance of artifact related APIs by adding indexes and refactoring sql logic

Closes #13890 #14813 #14814

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-20 11:25:43 +08:00
Wenkai Yin
c04f3a2aac Fix duplicate execution record issue
When the core service cannot response the checkin request in time, duplicated execution records may be created, this commit introduces the revision column to make sure there is only one record for one schedule trigger

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-05-12 15:40:52 +08:00
He Weiwei
476732df89 fix: change art_size to bigint in migrations
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-05-11 04:01:02 +00:00
Wenkai Yin
710c80078b Refactor the replication policy destination namespace logic
Support specifying what part of the repository will be replaced by the provided namespace

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-04-28 10:16:20 +08:00
Wenkai Yin
8b1817be0f Fix the consume too much CPU issue
1. Update execution status during the upgrade
2. Refine the execution sweeper

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-17 13:34:24 +08:00
Wenkai Yin
43df3bf8a4 Add upgrade sql file introduced in 2.1.4
1. Add upgrade sql file introduced in 2.1.4
2. Minor improvement for task/execution to cover corner cases

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-03-15 16:48:51 +08:00
Ziming Zhang
ec83f49a1a fix(retention) migrate sql error
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-02-05 09:09:26 +00:00
Wang Yan
24ec772978
fix gc migration issue (#14174)
For the upgrade path v1.10 - v2.1.* - v2.2.0, if user doesn't reset the GC schdule that was created in 1.10 in the v2.1,
the job parameters will keep empty in the database. The fix gives a default value for the schedule record.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-02-05 12:25:24 +08:00
Wenkai Yin(尹文开)
50a1e85095
Make sure the revision of execution isn't null during the upgrade (#14085)
Make sure the revision of execution isn't null during the upgrade
Fixes #14075

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-01-27 10:10:36 +08:00
Wenkai Yin
7c072e17a6 Fix the legacy scheduled job issue for GC/scan all
Fix the legacy scheduled job issue for GC/scan all

Fixes #13968

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2021-01-15 22:02:36 +08:00
He Weiwei
4a326aa8b0 chore: delete records of scan_report
The report in previous scan_report records not work well the
vulnerabilities stored in the schema table, so delete the scan_report
records.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2021-01-08 03:39:11 +00:00
Ziming Zhang
8faa76a1b6 feat(retention) refactor task manager
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2021-01-05 12:08:03 +08:00
prahaladdarkin
a890b28e1e
Store vulnerability data from scanner into a relational format (#13616)
feat: Store vulnerability report from scanner into a relational format

Convert vulnerability report JSON obtained  from scanner into a relational format describe in:https://github.com/goharbor/community/pull/145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
2020-12-25 08:47:46 +08:00
He Weiwei
3831e82b20
refactor: remove code of admin job (#13819)
Remove code of admin job as it's not needed by scan all/gc now.

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-22 11:48:16 +08:00
Wang Yan
9bc6f3cee4
fix robot account update issue (#13741)
* fix robot account update issue

enable the update method to support both v1 & v2 robot update

Signed-off-by: Wang Yan <wangyan@vmware.com>

* resolve review comments

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-18 20:01:26 +08:00
Wang Yan
6bc1047013
migration admin job data (#13766)
1, migrate gc and scan all schedule to schedule/task/exectuion
2, migrate gc history to task/execution

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-18 16:35:24 +08:00
Daniel Jiang
b0c8cadad7
Add default CVE allowlist to project library (#13770)
fixes #12700

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-12-16 14:20:56 +08:00
Wenkai Yin
69808f033e Tiny fixes for task manager
1. Add update time for execution
2. Add unique constraint for schedule to avoid dup records when updating policies
3. Format replication log
4. Keep the webhook handler for legacy replication jobs to avoid jobservice resending the status change request

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-12-14 17:26:32 +08:00
He Weiwei
08580f9fec
refactor(scan): refactor scan/scan all job to task manager (#13684)
Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-12-14 13:34:35 +08:00
Wang Yan
39e1a4f2b4 add extra attributes in the schedule table
Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-14 02:28:52 +08:00
wang yan
1bb79d402d update code per review comments
Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-10 16:08:52 +08:00
Wang Yan
dba5522d0b Migrate to task manager (#129)
1, remove the gc to new programming model
2, move api define to harbor v2 swagger
3, leverage task & execution manager to manage gc job schedule, trigger and log.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-12-10 14:00:33 +08:00
Wang Yan
d2fa2e6b84
update robot secret (#13654)
* update robot secret

1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object

Signed-off-by: Wang Yan <wangyan@vmware.com>

* update robot secret

1, use SHA256 to generate and validate robot secret instread of symmetric encryption.
2, update the patch input object

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-03 18:13:06 +08:00
Wang Yan
732e9a21cd
updates on robot accounts (#13623)
* updates on robot accounts

1, add patch method to refresh secret of a robot
2, fix robot account update issue
3, add editable attribute to handle the version 1 robot account
4, add duration for robot account
5, hide secret for get/list robot account

Signed-off-by: wang yan <wangyan@vmware.com>

* update code per review comments

1, change expirate creation func to AddDate().
2, remove the scanner duration specification, use the default value.

Signed-off-by: Wang Yan <wangyan@vmware.com>
2020-12-01 18:31:34 +08:00
DQ
907904f480 Add DB Migration code for clair cleanning
- Delete clair scanner if exist
- Delete report is it is scanned by clair
- Set Trivy to Default if it exist and not default scanner

Signed-off-by: DQ <dengq@vmware.com>
2020-11-29 16:19:02 +08:00
He Weiwei
eb38180483 fix(quota): ignore the fail when getting reference of quota
1. Clean the dirty data in quota/quota_usage.
2. Ignore the fail when getting the reference of quota.

Closes #13387

Signed-off-by: He Weiwei <hweiwei@vmware.com>
2020-11-24 14:50:38 +00:00
Wenkai Yin(尹文开)
fe8b628f0c
Merge pull request #13437 from ywk253100/200929_replication
Refactor the replication execution
2020-11-24 10:38:22 +08:00
Wenkai Yin
294385c34d Refactor the replication execution
1. Use the task manager to manage the underlying execution/task
2. Use the pkg/scheduler to schedule the periodical job
3. Apply the new program model
4. Migration the old data into the new data model

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-11-23 14:24:10 +08:00
stonezdj(Daojun Zhang)
fb549b2d9e
Merge pull request #13444 from wy65701436/robot2-self-mgr
add robot mgr
2020-11-16 11:33:33 +08:00